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Greg  Hutto,  53d  Test  Mgt  Group 

Jim  Simpson,  Florida  State  University,  Sverdrup 

□  Introductions 

□  Course  Overview 

□  Exercise 


2  Level  Factorial  Design 


Design  for  2  to  1 5  factors  where  each  factor  is  varied  over  2  ievels.  Usefui  for  estimating  main  effects  and  interactions.  Fractional  factorials  can  be  used  for 
screening  many  factors  to  find  the  significant  few.  The  color  coding  represents  the  design  resolution:  Green  =  Res  V,  Yellow=  Res  IV,  and  Red  =  Res  III. 
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Course  Overview 


□  Session  1  -  Central  challenge  of  test 

■  Deep  and  broad  testing 

■  What  is  DOE? 

■  History 

□  Session  2  -  Comparing  test 
strategies,  which  one  is  best? 

■  Conventional  logic 

■  Designed  experiment 


□  Session  3  -  Factorial  designs 

■  Phases  of  conducting  DOE  tests 

■  Project  decomposition 

■  Planning  the  test 

■  Producing  observations 

■  Pondering  results 

□  Session  4  -  Fractional  factorials  and 
other  topics 
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Course  Objectives 


□  Objectives: 

■  Understand  Test  Stats  through  ANOVA 

□  Plots  and  Displays 

□  Inferences  -  Sampling  Distributions 

□  Multi-way  experiments 

□  Fractional  Replication 

□  ANOVA  Extensions  and  Modifications 

■  Understand  classical  model  and 
assumptions 

□  Create  and  analyze  two-level 
experiments 

□  Detect  and  correct  violations  of 
assumptions 

□  Estimate  sample  size  and  formulate 
designs 

■  Be  able  to  use  Excel,  Resampling  and 
Design  Ease  for  Basic  Design  and  Analysis 
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Design  of  Experiments  --  15  Years 
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B.S.  US  Air  Force  Academy,  Operations 
Research 

M.S.  Air  Force  Institute  of  Technology, 
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Florida  A&M  -  Florida  State  University, 
Industrial  and  Manufacturing  Engineering 

Project  Experience 


□  CV-22 

□  53’''^  TMG  Consultant 

□  AGM  Simulation 

□  Wind  Tunnel  Testing 

□  WMD  Effectiveness 

■  Air  Force 

□  Army  BDA 

■  NASA 

□  Army  Operator 

■  NASCAR 

Effectiveness 
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Class  Introductions 


□  Name 

□  Degree(s)  +  Field  of 
Study 

□  Test  experience  - 

■  types  and  years 

□  Had  any  Stats? 

■  If  yes,  ever  used  it? 
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2  Level  Factorial 
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2  Level  Factorial  Design 


Design  for  2  to  1 5  factors  where  each  factor  is  varied  over  2  levels.  Useful  for  estimating  main  effects  and  interactions.  Fractional  factorials  can  be  used  for 
screening  many  factors  to  find  the  significant  few.  The  color  coding  represents  the  design  resolution:  Green  =  Res  V.  Yellow=  Res  IV,  and  Red=  Res  III. 
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The  SAM-in-a-Box  is  a 
potent  threat! 

Sold  to  Saddam  by 

Stat-a-Pult  Inc. 

We  must  conduct  an 
exploitation  HWIL 
simulation  today 

MOP  -  miss  distance 

Conditions  -  projectile, 
propellant,  Target  RCS  & 
velocity,  ECM  ...  etc 

To  start ...  Stopper 
Setting  @  2  levels 
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Planning  a  Test 


Objective:  Develop  an  experimental  design  for  understanding  the  effect  of  stop 
position  on  the  distance  of  a  projectile  travels  when  launched  from  a  catapult, 
hereafter  called  a  ‘statapult.’ 

Background:  Although  the  statapult  has  several  factors  that  may  affect  distance, 
we  will  only  consider  stop  position,  and  specifically  just  two  settings  for  stop. 

Your  task  will  be  to  design  a  test  that  will  capture  the  influence  of  changing  stop 
position  on  distance  traveled. 

Direction:  Develop  a  test  plan  for  determining  how  stop  position  affects  projectile 
launch  distance.  All  other  statapult  settings  will  be  specified  as  fixed.  Your 
mission  is  to  decide  the  number  of  tests  required,  the  order  in  which  to  run  the 
tests,  and  how  you  plan  to  assess  the  data  after  test.  You  have  10  minutes. 


#  of  Runs 


Run  Order 


Analysis  Approach 


Examp 


Run 
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e  or  Partial  Test  Matrix 


Stop  Setting 


Distance 
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Design 


Determine  the  effect  of  stop  position  on  distance. 


The  input  factor  is  stop  position.  Stop  position  will  have  two  settings. 


The  output  is  distance  in  inches.  Use  a  launch  angle  of  172  egrees. 


The  following  table  is  the  design  matrix.  Perform  the  experiments 
below. 


Run 


Distance 


pingle^Factor^D^ 
i 

M  Conduct  your  test  to  determine  the  effect  of  stop  position  on  distance.  The  first  four 
^  planning  steps  are  completed  below. 
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stop  Position 


Assignment: 


1. 


2. 


Each  team  will  have  a  launcher,  a  holder,  a  recorder  and  range  finders. 
Teams  are  not  permitted  to  communicate  with  other  teams. 

Collect  data  according  to  the  above  design  matrix.  Each  team  will  have 

3  minutes  to  organize  themseives  and  coiiect  the  data. 


3.  We  will  step  you  through  an  analysis  of  your  data. 
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Probability  Density  Function 
y=F(x.10.10) 


Probability  Distribution  Function 
p=iF(x.10.10) 
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Session  1  Testing  Deeply 
and  Broadly 


□  The  General  Test  Problem 

□  Testing  Deeply  -  1  Sample 

□  Review  of  Descriptive  Stats 

■  Basic  numerical  statistics 

■  Pictures  of  the  data 

□  Testing  Broadly  -  Spiral  1 

■  Design  for  Maverick  H/K 


2  Level  Factorial  Design 

Design  for  2  to  1 5  factors  where  each  factor  is  varied  over  2  levels.  Useful  for  estimating  main  effects  and  interactions.  Fractional  factorials  can  be  used  for 
screening  many  factors  to  find  the  significant  few.  The  color  coding  represents  the  design  resolution:  Green  =  Res  V,  Yellow=  Res  IV,  and  Red  =  Res  III. 


Number  of  Factors 


2  3  4  5  6  7  3  ^  10  11  12  13  14 


i 

i 

p 

p 

p 

p 

p 

p 

p 

p 

p 
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p 
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A  beer  and  a  blemish 


□ 

□ 

□ 

□ 


□ 


1906  -  W.T.  Gossett, 
a  Guinness  chemist 

Draw  a  yeast  culture 
sample 

Yeast  in  this  culture? 

Guess  too  little  - 
incomplete 
fermentation;  too 
much  --  bitter  beer 

He  wanted  to  get  it 
right 


□ 


□ 

□ 

□ 


□ 


sip  ^ 

5'd  • 


1998  -  Mike  Kelly, 
an  engineer  at 
contact  lens 
company 

Draw  sample  from 
15K  lot 

How  many 
defective  lenses? 

Guess  too  little  - 
mad  customers;  too 
much  —  destroy 
good  product 

He  wanted  to  get  it 
right 
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central  test  challenge  ... 


□ 


□ 


□ 
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IP' 

1  # 
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\  ' 
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j 

r 

In  all  our  testing  -  we  reach  into  the 
bowl  (reality)  and  draw  a  sample  of 
operational  performance 

Consider  our  "Sam-in-a-Box" 

■  Suppose  an  historical  70%  hit  rate 

■  Is  this  captured  version  at  least  as 
good? 

We  don't  know  in  advance  which 
bowl  God  hands  us  ... 

■  The  one  where  the  system  works  or, 

■  The  one  where  the  system  doesn't 
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Start  —  Blank  Sheet  of 
Paper 


□  Let's  draw  a  sample  of  n 
shots 

□  How  many  is  enough  to  get  it 
right! 

■  3  -  because  that's  how  much 
$/time  we  have 

■  8  -  because  I'm  an  8-guy 

■  10  -  because  I'm  challenged  by 
fractions 


■  30  -  because  something  good 
happens  at  30! 

□  Let's  start  with  20  and  see  .. 


I_3 

i 
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A  False  Positive  -  Declaring 
StataPult  is  Degraded  (when  it's 
not)  —  a 
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□ 


Suppose  we  fail 
StataPult  when 
we  get  50%  hits 

We'll  be  wrong 
(on  average) 
about  12%  of 
the  time 

We  can  tighten 
the  criteria  (fail 
on  60%)  by 
failing  to  field 
more  good 
weapons 

We  can  loosen 
the  criteria  (fail 
on  40%)  by 
missing  real 
degradations 

Let's  see  how 
often  we  miss 
such 

degradations  ... 
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In  this  bowl  - 
Statapult 
performance 
meets  spec 
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A  False  Negative  -  Declaring 
Statapult  Meets  Spec  (when  it 
doesn't)  — 
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□  Use  the  failure 
criteria  from  the 
previous  slide 

□  If  we  field 
Statapult  with 
55%  hits,  we  fail 
to  detect  the  true 
degradation 

□  If  Statapult  fails 
spec,  with  n=20 
shots,  we'll  miss 
it  about  80%  of 
the  time 

□  We  can,  again, 
tighten  or  loosen 
our  criteria,  but 
at  the  cost  of 
increasing  the 
other  error 


In  this  bowl  - 
Statapult  Prob(in) 
decreased  10% 

from  70%  to  60% - 
it  is  degraded 


0.15  0.25  0.35  0.45  0.55  0.65  0.75  0.85  0.95  1.05 
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Q1  -  How  Many  Runs? 
Testing  Deeply  Enough  to 
Balance  Our  Chance  of  Errors 
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□  Putting  them 
together,  we  see 
we  can  trade  one 
error  for  the  other 
(a  for  P) 

□  We  can  also 
increase  sample 
size  to  decrease 
our  risks  in  testing 

□  These  statements 
are  not  opinion  - 
they  are 

mathematical  fact 

and  an  inescapable 
challenge  in  testing 

□  There  are  two 
other  ways  out ... 
factorial  designs 
and  real-valued 
MOPS 


Getting  it  right:  Conf7cfence\n  stating  results; 
Powerto  find  small  differences 


AF  T&E  Days  -  Dec  05  1-22 


Design  of  Experiments 


¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 


Q1  -  How  Many  Runs? 


■-0  ^  i’ 


□  We  teach  our 
analysts  and 
test 

engineers  to 
custom- 
design 
sample  size 
for  each  test 

□  The  answer  is 
-  it  depends 
on: 

■  Appetite 
for  risk 

■  Difference 
we  care 
about 

■  Amount  of 
noise  in 
the  system 
under  test 


The  General  Answer 
-  Power  curve  for 
this  test 


Degradation  CE50  to  (BO 
alpha = beta  =  10% 
betaerrorrate:.10000 


.32  .34  .36  .38  .4  .42  .44  .46  .48  .5 

Degradation  from  CE50  to  (I_ 
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Recap  -  First  Two  Challenges 


□ 


□ 


Challenge  1:  effect  of  sample  size 
on  errors  -  Depth  of  Test 

Challenge  la:  measuring  best 
response  -  rich  and  Relevant 


□  So  --  it  matters  how  many  we  do 
and  it  matters  what  we  measure 

□  There  is  a  2nd  challenge  -  Breadth 
of  testing  -  searching  the 
employment  battlespace 
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SAM-in-a-Box 
HWIL  ..  continued 
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step  2.  Now  add  delivery  method  (arm  draw¬ 
back  distance)  and  target  RCS  (stealth  or 
conventional)  to  the  test  conditions.  Revise 
your  test  matrix  and  reconsider  the  previous 
questions  (trials,  order,  analysis) 

You  might  be  interested  in  answering  questions 
like: 

1.  What  conditions  produce  the  best  accuracy 
on  the  target? 

2.  What  conditions  are  robust  to  uncontrollable 
factors  (jamming,  fatigue,  chaff,  haze)? 

3.  What  conditions  fail  to  make  a  difference  in 
accuracy? 

With  these  three  test  conditions,  how  will  you 
revise  your  test  to  answer  the  above 
questions? 

This  situation  is  exactly  that  faced  by  our 

test  teams.  In  this  course,  we  will  learn 
about  a  method  to  answer  them... 
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Spiral  1  —How  We  Design 
Tests  in  4  Stages 
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Ponder 


I  Project  Description  and 
Decomposition 

II  Pian  the  Test  Matrix 

III  Produce  the  Observations 

IV  Ponder  the  Resuits 
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We’ll  stroll  quickly  through 
examples  of  each  step  for  Maverick 

Problem 


Switch  Examples:  The 
Maverick  H/K 


□  Present  B  (EO)  and  D/G  (HR)  seekers  are 
degrading  with  age 

□  Replace  EO/IR  versions  with  new  H  and  K 
variants 

□  Critical  Operational  Issue:  Does  Maverick 
still  perform  at  least  as  well  as  previously? 

□  For  our  purposes  -  we  shoot  several 
Mavericks  of  each  type  -  same  day,  same 
conditions,  same  target 

□  What  are  our  chances  of  getting  it  right? 
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Challenge  2:  Broader  --  How 
Do  Designed  Experiments 
Solve  This? 
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Designed  Experiment  (n). 
Purposeful  control  of  the  inputs 
(factors)  in  such  a  way  as  to 
deduce  their  relationships  (if  any) 
with  the  output  (responses). 


¥ 


Maverick  K  or  G 


Slew  Sensor  (TP,  Radar) 
Target  Type cn 


Platform  (F-15E,  F-  16C) 


Shooting 

Maverick 

Missiles 


RMS  Trajectory  Dev 


Hits/misses 


P(damage) 


Miss  distance  (m) 


nputs  (Conditions) 


Outputs  (MOPs) 


G.E.P  Box  (son-in-law  to  Fisher)  - 
"All  math  models  are  false  ... 
but  some  are  useful." 

"All  experiments  are  designed  ... 
most,  poorly." 
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The  General  Test  Problem  -  A 
p  Little  More  Complicated  Black 
P  Box 

i 

Blue  Controllable  Faetors  (x) 
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Test 

Controllable 
Factors  (w)  ^ 


Process  or  System 
under  Test 


Outputs 

(y) 


Environmental/Uncontrollable  Covariates  (z) 
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□  Questions  that  must  be  answered  for  any  test: 

■  Which  of  the  variables  (x's,  w's)  to  change,  what  pattern, 
what  range?  How  deal  with  z's? 

■  How  many  runs  should  we/can  we  make? 

■  How  should  the  runs  be  sequenced? 

■  What  method  of  analysis  should  be  employed? 

□  Possible  Objectives: 

■  Characterizing  where  y's  usually  fall  depending  on  x  settings 

■  Determining  which  variables  have  or  do  not  have  an 
influence  on  outputs 

■  Determining  where  to  set  influential  variables  so  outputs 
are  near  goal 

■  Determining  where  to  set  x's  so  that  effects  of  z's  are  small 

Answers  determine  your  strategy  of 
experimentation. 
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PROCESS  Process  Flow: 

I  Steps  in  Black  Box  -  MOPs 
to  measure 
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Mission  Planning"^ 


Ground  Checks 


Flight  Checks 


Time 

Success 

Quality 


Success 

Time 

Effectiveness 

Faults 


Success 
Time 
Effectiveness 
Faults 


Attacks 


Target 

Acquisition 


Time  (source) 
Desig  Error 


IP  Accuracy 
Time  to  IP 
Detect  Tgt 


Flight  profile 
Weapon  guide 
Impact  point 


Track  Accurac)' 
Break  Locks 
Reacq  time 


Accurate  Cues 
Success 


□  The  test  team  (Ops,  Analyst,  PM, 
eng)  breaks  process  down 

■  What  are  the  events,  steps, 

outcomes? 

■  What  are  the  ops  choices  and 
conditions  at  each  step? 

■  How  do  we  measure  success? 

□  Result  -  Results  to  Measure 
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Cause  and  Effect  Diagrann 
to  Elicit  Maverick  Shot  Test 
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DOE* 


Measurement 

5 

Manpower 

Materials 

Cause  Labels: 

_  C  —  Constant  (run  card) 

_  N  —  Noise  no  control 
X  —  Control  Variable 

Rng  (kill) 


1  ^ 

Variable 

Units 

Range 

Priority 

Ekp.  C^trol 

Design  Range 

1 

Target 

Qual 

many 

H 

X 

trk,  tank,  bid 

2 

Platform 

Qual 

F-15/16 

H 

X 

F-15,  F-16 

3 

Time  of  Day 

hour 

0-24 

H 

X 

Dawn,  Noon 

4 

Humidity 

gm/m3 

30-Jun 

H 

X 

Nellis  Eglin 

5 

Tgt  Velocity 

mph 

0-40 

H 

c 

stationary 

6 

Operator  Skill 

Qual 

L,M,H 

M 

c 

H-  captive  mns 

7 

Mav  Model 

Qual 

BDGHK 

L 

c 

B-HorD-K 

8 

Wind 

m/s 

0-10 

L 

N 

measure,  R 

¥ 
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Battlespace  Measures  & 
Conditions  for  Maverick  H/K  Case 
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Type 


Objective 


Measures  of 
Performance 


Subjective 


Measure  of  Performance 


Target  acquisition  range 


seeker  lock-on  range 


launch  range 


mean  radial  impact  distance 


probability  of  target  impact 


reliability 


Interoperability 


human  factors 


tech  data 


support  equipment 


tactics 


Conditions 

Settings 

#  Levels 

Missile  Variant: 

H,  K,  B,  G 

4 

Launch  Platform: 

F-i6c,F-i5E,A-ioA  Test  Cooditions 

3 

Launch  Rail: 

LAU-117,LAU-88 

2 

Target: 

Point,  Area 

2 

Time  of  Day: 

Dawn/Dusk,  Mid-Day 

3 

Environment: 

Forest,  Desert,  Snow 

3 

Weather: 

Clear  (+7nm).  Haze  (3-7nm),  Low  Ceiling/Visihility  (<3000/3nm) 

3 

Humidity: 

Low  (<30%),  Medium  (31-79%),  High  (>80%) 

3 

Attack  Azimuth: 

Sun  at  back.  Sun  at  beam.  Sun  on  nose 

3 

Attack  Altitude: 

Low  (<5000’),  High  (>5000’) 

2 

Attack  Airspeed: 

Low  (Mach  .5),  Medium  (Mach  .72)  ObviOUSly 

3 

Missile  Mode: 

Centroid  track.  Force  correlate  track  3  laro© 

2 

Combinations 

^  139,968 

¥ 

¥ 

¥ 

¥ 


envelope 
. . .  how  to 
search  it? 
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PLAN:  Algorithm  to  Construct 
Full  Factorial  Run  Matrices 


□  How  many  Variables  (A,B,C  ...  K)? 

□  How  many  settings  (levels)  for  each  Variable  (a,  b, 
c,  ...  k)? 

□  How  many  runs  total? 

■  a  *b*  ...  *k 

■  Example:  4  vars:  1  at  3,  3  at  2  levels 

■  Runs  (N)  =  3*2*2*2  =  3  *  2^  =24  runs  (test  cells) 

□  Construct  Design  — 

■  Var  A  set  at  1  for  N/a,  level  2,  etc. 

■  With  A  at  1,  set  B  at  1  for  N/a/b,  at  2  for  N/a/b... 

■  Continue...  Show  on  Board 
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I  Coded  Formulation  Example  - 

I  -  Factorials  (Solution) 

¥  ^ 


Case  MavVer  Target  T.O.  Day 


3 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 


1 


-1 


-1 


-1 


2 

-1 

-1 

1 

3 

-1 

1 

-1 

4 

-1 

1 

1 

5 

1 

-1 

-1 

6 

1 

-1 

1 

7 

1 

1 

-1 

8 

1 

1 

1 

Mav  Ver 


Target 


□  Note  the  geometric  shape  you  built 

□  All  variables  are  orthogonal  (right 
angles) 

□  General  algorithm  to  build  --  vary  A  for 
half  of  runs,  B  for  half  of  A  and  C  for  half 
of  B  ... 
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Challenge  2:  Systematically 
Search  the  Relevant 
Battlespace 
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4  reps  1  var 


i  □ 

I 

i 

i 

I  □ 
i 
i 
i 

i 
I 
I 


□ 


Factorial  (crossed) 
designs  let  us 
learn  more  from 
the  same  number 
of  assets 

We  can  also  use 
Factorials  to 
reduce  assets 
while  maintaining 
confidence  and 
power 

Or  we  can  combine 
the  two 


w 


All  four  Designs  share 
the  same  poiverand 

confidence 


i 

p 

p 

AGM-65D 

AGM-65K 

onr — 

-N, 

4 

4 

V 

2  reps  2  vars  AGM-65D 


Truck 


Tank 


2 


2 


AGM-65K 


2 


2 


1  reps  3  vars 

AGM-65D 

AGM-65K 

Eglin  (Wei) 

Truck 

1 

1 

Tank 

1 

1 

Nellis  (Diy) 

Truck 

1 

1 

Tank  □ 

1 

1 

V2  rep  4  vars 


Dawn 

(Transition) 


Midday 

(Stable) 


Eglin  (Wet) 


Nellis  (Dry) 


Eglin  (Wet) 


Nellis  (Dry) 


Tank 


Truck 


Tank 


Truck 


Tank 


Truck 


Tank 


AGM-65D 


AGM-65K 


i 
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M  Way  Designs 


MWay 

Response 

Surface 

Designs 


emplates 


PLAN:  With  DOE  we 

1*  customize  matrices  from  a 

¥ 

¥ 

¥ 

¥ 

¥ 

P  One  Way 
P  With  Blocks 


Design  of  Experiments 

y  ^  ^  It  ' 


B1  1 

B12RCBD 


•  • 

Central  Composite 


'  V  ^ 

*  ' 


Box-Behnkin 


(And  Blocks) 
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PRODUCE  data  with  Random 
run  order 


Design  of  Experiments 

y  ^  *  It  ' 


□ 


Randomizing  runs  protects  from 
unknown  background  changes 
within  an  experimental  period 
(due  to  Fisher) 
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PRODUCE  with  Blocks 


□ 


Day  1 

/\jN< 

. 

.•7  : 

••/  • . •  :  de 

/  : 

y 

• 

••••••••• 

tectio 

Day  2 

. 

.*■  *•..  /  ..V* 

n\(no  hjSb^s) 

/  . 

'  f  \ 

V  /  \ 

1  detection  (bio 

L  .....  . 

cks) 

\ 

[ 

uence 

•  •  •  •  i 

O  O  O  C)  : 

I  rur 

”  ”  ” ~  ~  ~ 

_ 1 

1 

1  ^PP 

1 

O  large  target 
•  small  target 


Blocking  designs  protects  from 
unknown  background  changes 
between  experimental  periods  (also 
due  to  Fisher) 
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PONDER  Simple  Analysis  — 
How  Conditions  Affect  MOP 


1>' 


^^"+(01(1-0) 


-  (New- 
+  (Dawn) 


Time  (B) 


-  (Noon) 


bc=ll 


b=19 


ab=17 

Miss  Distance 
measured  at 
each  condition 


^c=9  afc=ll 

^  (1)  =  25  a=24 

-  (Tank)  +  (Truck) 

Target  (A) 


□  8  Runs  -  Solve  for  8  Unknowns: 

□  Overall  Mean  --  Effect  (Target)  --  Effect  (Time)  — 
Effect  (Maverick) 

□  +  3x2-ways,  3-way  interactions 


Designed  Experiment 

(n).  Purposeful  control 
of  the  inputs  (factors)  in 
such  a  way  as  to  deduce 
their  relationships  (if 
any)  with  the  output 
(responses). 
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AC 

BC 

ABC 
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Actual  Maverick  H/K  Test 
-  A  great  success 
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Mission:  MAV-1 

Eglin  Range  Range  Time:  0700-0900  (Dawn)  Target:  Point  (Tank)  Weather:  As  Scheduled 

Launch  Airspeed:  400KIAS 

F-16-#l 

Left  Wing: 

CATM-65Kon 

LAU-88 

Right  Wing:  T( 

M-65B  on  LAI 

J-88 

Run# 

Target  Type 

Altitude 

Sun  Angle 

Missile  Type 

Cueing 

Comments  * 

1 

Tank/Tmck 

18,000’ 

Sun  at  6 

H/K 

Visual 

2 

Tank/Tmck 

18,000’ 

Sun  at  6 

B 

1 

3 

Tank/Tmck 

1500’ 

Sun  at  3/9 

H/K 

NAV/CPS 

1 

4 

Tank/Tmck 

1500’ 

Sun  at  3/9 

B 

1 

5 

Tank/Tmck 

1500’ 

Sun  at  6 

H/K 

Radar 

1 

6 

Tank/Tmck 

1500’ 

Sun  at  6 

B 

1 

7 

Tank/Tmck 

18,000’ 

Sun  at  3/9 

H/K 

LANTIRN 

1 

8 

Tank/Tmck 

18,000’ 

Sun  at  3/9 

B 

F-16- #2 

Left  Wing:  CATM-65K  on  LAU-88  Right  Wing:  TCM-65D  on  LAU-88 

1 

Run# 

Target  Type 

Altitude 

Sun  Angle 

Missile  Type 

Cueing 

Comments  * 

1 

1 

Tank/Tmck 

18,000’ 

Sun  at  6 

H/K 

Visual 

1 

2 

Tank/Tmck 

18,000’ 

Sun  at  6 

D 

1 

3 

Tank/Tmck 

1500’ 

Sun  at  3/9 

H/K 

NAV/GPS 

1 

4 

Tank/Tmck 

1500’ 

Sun  at  3/9 

D 

1 

5 

Tank/Tmck 

1500’ 

Sun  at  6 

H/K 

Radar 

1 

6 

Tank/Tmck 

1500’ 

Sun  at  6 

D 

1 

7 

0 

Tank/Tmck 

'  1  ’ r\  ^  At-!'  I  ’ “TH  7- 

18,000’ 

1  0  nnn’ 

Sun  at  3/9 

a/o 

H/K 

LANTIRN 

After  simulated  pickle,  simulate  missile  flyout  Typical  Mav  H/K  F-16  Run  Card 

Extensive  captive  carry 

■  22  sorties  -Approx  100  sim  shots 

■  Old/new  seekers  on  each  wing  to  equalize  Wx 

■  3  platforms:  F-16,  F-15E,  A-10 

■  Eglin  &  Nellis 

Results  -  approx  2x  acq/trk  range 
9  shots  comparable  to  current  performance 
Type  III  Error  -  looking  in  the  wrong  place  ... 
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A  little  bit  of  History 


□  DOE  has  roots  in  Agriculture,  Manufacturing 

■  Royal  Agricultural  Experiment  Station  (R. 
Fisher  -  1920's) 

■  Industrial  Process  Control  (Deming,  Shewart  - 
-  1930's-40's) 

■  Quality  Revolution  in  Industry  &  Government 
(AFIT,  NASA,  Edwards,  Ford,  Motorola) 

□  Recent  Developments  include: 

■  Fractional  factorials  (1960's) 

■  Optimal  Design  of  Experiments  (1980's) 

■  Measuring  dispersion  effects  (1990's) 

□  People  use  DOE  because  it  works  ... 

■  Highly  efficient 

■  Unambiguous  results 

■  Long  track  record  in  science,  engineering, 
medicine  and  industry 

□  DOE  is: 

■  A  Test  PROCESS  Philosophy 

■  A  Collection  of  Test  Matrix  DESIGN  Templates 

■  A  Set  of  ANALYSIS  Methods 

AF  T&.E  Days  -  Dec  05 


¥ 

¥ 

¥ 

¥ 

I 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 


Historical  DOE  Timeline^ 


1800 

1850 

1900 

1910 

1920 

1930 

1940 

1950 

1960 

1970 

1980 

1990 

2000 


Least  Squares  (Gauss,  Legendre) 


Regression  Concepts  (Person,  Galton) 


t-test  (Gosset) 


Factorial  Designs 
and  ANOVA  are 
DOE.  DOE  was 
first  developed  and 
used  in  crop  trials 
by  Sir  R.  A.  Fisher, 
a  mathematician 
and  geneticist 

_ ^ 


Factorial  Experiments  and  ANOVA 
(Fisher) 

Formalized  Hypothesis  Tests  (Neyman,  Pearson) 

2^  Factorial  Designs  (Yates) 

Fractional  Factorial  Designs  (Finney,  Rao) 

2k-p  Fractional  Factorial  Resolution  (Box,  Hunter) 

Taguchi  develops  his  Methods 

Central  Composite  Designs  (Box,  Wilson) 

Optimal  Designs  (Kiefer,  Wolfowitz) 

Box-Behnken  Designs  (Box,  Behnken) 

Algorithm  for  D-optimal  designs  (Johnson,  Nachtsheim) 
Detecting  dispersion  effects  ratio  of  variances  (Montgomery) 


1.  Source:  Appendix  K  — 
Understanding  Industrial  Designed 
Experiments, 

Schmidt  and  Launsby,1998 
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Patron  Saint  of  DOE  - 
Sir  R.A.  Fisher 
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Sir  Ronald  Aylmer  Fisher 


http://www-historv.mcs.st-andrews.ac.uk/historv/Mathematicians/Fisher.html 

Born:  17  Feb  1890  in  London,  England 
Died:  29  July  1962  in  Adelaide,  Australia 

Ronald  Fisher  received  a  B.A.  in  astronomy  from  Cambridge  in  1912.  There  he  studied 
the  theory  of  errors  under  Stratton  using  Airy's  manual  on  the  Theory  of  Errors  .  It  was 
Fisher's  interest  in  the  theory  of  errors  in  astronomical  observations  that  eventually  led 
him  to  investigate  statistical  problems. 

Fisher  gave  up  being  a  mathematics  teacher  in  1919  to  work  at  the  Rothamsted 
Agricultural  Experiment  Station  where  he  worked  as  a  biologist  and  made  many 
contributions  to  both  statistics  and  genetics.  He  had  a  long  dispute  with  Pearson  and  he 
turned  down  a  post  under  him,  choosing  to  go  to  Rothamsted  instead.  There  he  studied 
the  design  of  experiments  by  introducing  the  concept  of  randomisation  and  the  analysis 
of  variance,  procedures  now  used  throughout  the  world. 

In  1921  he  introduced  the  concept  of  likelihood.  The  likelihood  of  a  parameter  is 
proportional  to  the  probability  of  the  data  and  it  gives  a  function  which  usually  has  a 
single  maximum  value,  which  he  called  the  maximum  likelihood. 

In  1922  he  gave  a  new  definition  of  statistics.  Its  purpose  was  the  reduction  of  data  and 
he  identified  three  fundamental  problems.  These  are  (i)  specification  of  the  kind  of 
population  that  the  data  came  from  (ii)  estimation  and  (iii)  distribution. 

The  contributions  Fisher  made  included  the  development  of  methods  suitable  for  small 
samples,  like  those  of  Gosset,  the  discovery  of  the  precise  distributions  of  many  sample 
statistics  and  the  invention  of  analysis  of  variance.  He  introduced  the  term  maximum 
likelihood  and  studied  hypothesis  testing. 

Fisher  is  considered  one  of  the  founders  of  modern  statistics  because  of  his  many 
important  contributions.  He  was  elected  a  Fellow  of  the  Royal  Society  in  1929,  was 
awarded  the  Royal  Medal  of  the  Society  in  1938  and  he  was  awarded  the  Darwin  Medal 
of  the  Society  in  1948:-  in  recognition  of  his  distinguished  contributions  to  the  theory  of 
natural  selection,  the  concept  of  its  gene  complex  and  the  evolution  of  dominance. 

Then,  in  1955,  he  was  awarded  the  Copley  Medal  of  the  Royal  Society:-  in  recognition  of 
his  numerous  and  distinguished  contributions  to  developing  the  theory  and  application  of 
statistics  for  making  quantitative  a  vast  field  of  biology. 
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He  was  awarded  the  Royal  Medal  of  the  Royal  Society  in  1938  and  the  Copley  Medal  in 
1955.  You  can  see  a  history  of  the  Royal  Medal  and  a  list  of  the  winners  in  our  archive 
and  a  history  of  the  Copley  Medal  and  a  list  of  the  winners. 
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A  Strategy  to  be  the  Best 
...  In  Test  Design 
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DOE* 


□  Train  53d  Wing 
Leadership  in  statistical 
thinking  for  Test 

□  Adopt  the  most 
powerful  test  strategy 
(DOE) 

□  Train  total  team  from 
PM  to  OA 

□  Develop  mentors  and 
formally  mentor  test 
teams 

□  Revise  Wing  procedures 

□  Share  these  test 
improvements 


Adopters 

Tarqets 

■  AFOTEC 

■  HQAFMC 

■  18  FTS 

■  AAC 

(AFSOC) 

■  46TW 

■  F-15  0FP 

CTF 

■  USAFTPS 

■  AFFTC 
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The  /.east  You  Should 
Recall... 
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□  The  Central  Problem  of  Test  is  inferring  what 
the  real  world  is  like  based  on  our  sample 

■  We  can  make  one  of  two  errors  -  false  positive 
and  false  negative 

■  Want  to  design  our  tests  to  minimize  these 
risks 

□  A  Designed  Experiment  is  about  the  Design  - 
the  pattern  of  test  conditions  we  run 

■  Two  variabies  are  Confounded  when  we  cannot 
separate  their  effects  on  the  MOP  (response) 

■  DOE  avoids  confounding  two  or  more 
variables 

□  Basic  process  to  Construct  is  Process-Plan- 
Produce-Ponder 

□  DOE  has  80-year  track  record  in  all  areas  of 
science,  engineering,  and  test 

□  DOE  is  a  test  strategy  in  the  test  big  picture 

□  DOE  has  made  53rd  Wing  tests  better,  faster 
and  cheaper 

□  Resampling  (Monte  Carlo)  statistics  give  us  a 
powerful  tool  to  directly  attack  hard  problems 
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Resources 


□  Links: 

https  ://wwwmil.53wq.eQlin.af.  mi 

l/mil  web/index,  htm  —  Training 
menu  (53d  Wing  Courses) 

□  www.statsoft.com  (Statistica) 

□  www.minitab.com  (MiniTab) 

□  www.statease.com  (Design  Ease) 

□  www.resample.com  (Resampling 
Stats) 

□  Books: 

■  Design  and  Analysis  of  Experiments, 
5th  Ed  Douglas  Montgomery,  2001 

■  Statistics  for  Experimenters,  Box, 
Hunter  &  Hunter,  1978 

■  Design  of  Experiments:  Statistical 
Principles  of  Research  Design  and 
Analysis,  Robert  Kuehl 
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I  Session  2  -  Comparing  Test 
p  Strategies 

8*  ^ - 

P 

p  □  Is  DOE  the  Best  Test  Strategy? 

P  □  Other  Strategies 
P  ■  One  factor  at  a  time 

P  ■  Previous  experience 

S  ■  Scenario  testing 


at  □  Interactions 


p 

p 

p 

p 

p 
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What's  Best?  Many  methods 
of  test  have  been  tried 
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□  Our  next  example  will  characterize  the  B- 
IB  Radar  Target  Location  Error  as  a 
function  of  several  factors 


Altitude 


Target  RCS 


Angle  off  nose 
Aircraft  tail  number 
Target  Range 
Radar  calibration  datel 

Last  doppler  update 
Target  Elevation 

Operator  Skill  Level 

Inputs  (test 
conditions) 


Process: 

Radar 

Ground 

Mapping 


Target  location  accuracv 


Outputs  (MOPs) 


□  Previous  experience 

□  Contractor  generated 

□  Intuition 

□  One  factor  at  a  time  (OFAT) 

□  Scenario  (best  guess  or  case) 
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B-1  Radar  Target  Location 
Error  Test  Space  Magnitude 


□  How  big  is  the  test  space? 

□  Consider  9  factors  minimum  - 
two  levels  each 

■  For  example,  set  angle  off  nose  to 
15  or  30  degrees 

□  If  we  tested  each  possible 
combination,  how  many  would 
there  be? 


29  =  512 


How  should  we  examine 
this  test  space? 
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I  BTW  ...  the  "Plus  2  Ways 


|l  □  Previous  experience 
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■  Picture  a  young  ILt  assigned  as  Lead 
Analyst,  F-16C  Block  0 

■  Q:  What  will  he  do? 

■  A:  The  best  he  can  ... 

■  Graybeard  advice  -  get  the  F-16A 
MOT&E  Test  Plan! 

■  Obviously  -  not  a  general  solution  ...  how 
good  is  F-16A  Test?  What  about  Global 
Hawk  and  JSF? 

Run  contractor-selected  test  points 

■  Current  53d  Wing  CTF  Test  -  run  these 
227  cases  3X  each 

■  Q:  Why  these  points?  Why  3  times? 

■  A:  Because  that's  what  we  did  last  time. 
We  do  3  to  get  1. 

■  Q:  How  do  you  compare  new  OFP  to  old 
OFP  performance? 

■  A:  Mostly  we  just  look  at  them. 


□  When  we  look  under  covers  ... 
mostly  one  of  three  strategies 
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Intuition 


□  One  way  to  examine  the  test 
space  is  to  reiy  on  intuition 

□  Intuition  can  greatly  benefit  a 
test;  however... 

■  Requires  deep  subject  matter 
knowledge  that  is  not  always 
available 

■  Typically  only  "discover"  what  you 
already  believe  to  be  true 

■  Lacks  proof 

□  Intuition  can  assist  with  test 
design  but  it  should  never  be 
the  sole  strategy 
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OFAT  Design  looks  like 
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The  Variables  (Factors)  "“I’?®''  combinations^  ? 

At  two  levels  each  variable 

A.  Angle  off  nose  have  2x2x2...  =  2^  =  512 

B.  Range  to  target  combinations 

C.  Clutter  background 

D.  Taraet  RCS 

E.  Aircraft  velocity 

F.  Aircraft  altitude 

G.  Years  OSO  experience 

H.  Time  since  radar  calibration 
J.  Time  since  doppler  update 


110  total 
resources 
usei 


|Case 

A 

B 

C 

D 

E 

F 

G 

H 

J 

Y1 

"T 

Y10| 

I  Avg 

1  ^ 

low 

low 

low 

low 

low 

low 

low 

low 

low 

\ 

1  2 

hi 

low 

low 

low 

low 

low 

low 

low 

low 

T 

1  ^ 

best 

hi 

low 

low 

low 

low 

low 

low 

low 

1  ^ 

best 

hi 

low 

low 

low 

low 

low 

low 

1  ^ 

best 

hi 

low 

low 

low 

low 

low 

r 

1  6 

best 

hi 

low 

low 

low 

low 

We  ’re  going  to  run 
10  because  we  ’re 

1  7 

best 

hi 

low 

low 

low 

1  ^ 

best 

hi 

low 

low 

10 -guys 

and  that ’s 

1  ^ 

best 

hi 

low 

what  we  do. 

1  ^0 

best 

hi 

1  11 

best 

P 

P 

P 

P 

P 


□  11  combinations  tested 

□  No  math  model  to  predict  other  501  combos 

□  No  ability  to  estimate  if  variables  interact 
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Yesterday's  typical  OFAT 
E  Test  -  Brewing  Beer  ... 
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Objective  is  to  maximize  "beer  goodness 


ff 


ouaiici  piui 
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□  OFAT  works  if  response  contours  align  with 
axes...  but  what  if  the  contours  are  not 
aligned? 
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L  OFAT  Assumption 


i 

J  If  contours  are  not  aligned  with  axes,  we  miss  the 
P  optimum. 

P  Problem  is  not  simply  to  find  best  time  and  then  best 
temperature  —  we  must  find  if  the  variables 

interact 

(Response  contours  are  often  ridges,  saddles  and 
other  shapes  at  an  angle  to  our  control  axes) 


¥ 
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Orthogonal 
DOE  Factorial 
n 


ofo 
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Back  to  B-1  Radar:  the 
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I  meaning  of  interaction 
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Lateral  Error  vs.  Angle  and  Range 


_ 15  deg  off  nose 

—  30  deg  off  nose 


Near 


Range  to  Target 


Far 


Parallel  slopes  of  A  with  B  — 
response  at  level  of  B  does  not 
depend  on  level  of  A 

Also  —  can  talk  about  variable 
settings  independently  —  "15 
degrees  off  nose  always  best" 

Simple  model: 

Y=bo-i-bi*Range-i-b2*Angle 


Nonparallel  slopes  of  A  with  B 
—  response  at  level  of  B  does 
depend  on  level  of  A 

Also  —  must  talk  about 
variable  settings  together  — 
"Long  range  targets  detected 
best  at  15  degrees  off  nose" 

More  complex  model  with 
interaction  term: 

Y=bo-i-bi*  Range  -i-b2*  Angle 


+ 


bn*  Range  *  Angle 


50 


LU 

fu 

1- 

(D 
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Lateral  Error  vs.  Angle  and  Range 


Near 


Range  to  Target 


_ 15  deg  off 

nose 

— —  30  deg  off 
nose 

Far 


OFAT  cannot  detect  interaction 
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Best  Guess  or  "Cases"  Testing 
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|i  relies  on  intuition  (not  fact) 
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Case 

A 

B 

C 

D 

E 

F 

G 

H 

J 

1 

low 

hi 

low 

hi 

low 

hi 

low 

low 

low 

2 

hi 

hi 

low 

low 

low 

low 

low 

low 

low 

3 

low 

low 

hi 

low 

hi 

low 

low 

low 

low 

4 

low 

low 

low 

low 

low 

hi 

low 
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low 

5 

low 

low 
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low 

low 

low 

low 

hi 

hi 

6 

low 

low 

low 

low 

low 

hi 

hi 

low 

low 

7 

low 

low 

low 

low 

low 

low 

low 

low 

low 

8 

low 

low 

hi 

hi 

hi 

low 

low 

low 

low 

9 

low 

hi 

low 

low 

hi 

low 

low 

low 

low 

10 

11 

low 

hi 

low 

hi 

Best  guess  means  choosing  th 

Y1 


Y17 


Avg 
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Usually,  the  organization  has  a  magic  number  of 
replications  (3,8,30...)  they  believe  will  be  "good"  in 
some  unspecified  sense.  Often  they  say  "significant' 
"data  is  normal." 


or 


Case 

A 

B 

c 

D 

E 

Ybar 

1 

low 

hi 

hi 

low 

low 

20 

2 

hi 

low 

low 

hi 

low 

9 

3 

low 

hi 

hi 

low 

low 

3 

4 

low 

low 

low 
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low 

8 

5 

hi 

hi 

low 

low 

low 

7 

6 

low 

hi 

hi 

low 

low 

8 

7 

low 

low 

hi 

hi 

hi 

24 

8 

low 

low 

hi 

low 

low 

16 

9 

low 

hi 

low 

hi 

low 

2 

10 

hi 

hi 

hi 

hi 

hi 

30 

11 

low 

low 

low 

low 

low 

6 

If  runs  1-11  were  best 
guess,  what  can  we 
conclude? 

1.  Does  factor  E  affect 
the  average? 

2.  Can  we  separate 
effects  of  B  and  C? 

3.  What  D,  E  combo  is 
missing? 
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Scenario  Designs  - 
Confounding  among  Predictor 


Matrix  of  Correlation  Coefficients  among  X  Variables 


k 


B 


k 


B 


□  Such  correlations  make  independent 
estimates  of  effects  impossible 

□  In  English  -  the  effects  of  the  test 
conditions  are  muddled:  low  effects  are 
magnified  and  high  effects  are  reduced 

□  And,  we  cannot  detect  when  two 
conditions  work  together  to  change  the 
response  (interactions) 
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Maverick  example-DOE  Crosses 
l^factors  to  avoid  confounding  and 
l*estimates  effects  and  interactions 
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4  reps  1  var 


( 

AGM-65D 

AGM-65K 

onr 

2 

( 

4 

4 

2  reps  2  vars  AGM-65D 


Truck 


Tank 


2 


2 


□ 


□ 


f 

f  □ 


Factorial  (crossed) 
designs  let  us  learn 
more  from  the 
same  number  of 
assets 

We  can  also  use 
Factorials  to  reduce 
assets  while 
maintaining 
confidence  and 
power 

Or  we  can  combine 
the  two 


AGM-65K 


2 


2 


1  reps  3  vars 

AGM-65D 

AGM-65K 

Eglin  (Wei) 

Truck 

1 

1 

Tank 

1 

1 

Nellis  (Diy) 

Truck 

1 

1 

Tank  □ 

1 

1 

V2  rep  4  vars 


All  four  designs  share  the 
same  poiverand 
confidence 


Dawn 

(Transition) 


Midday 

(Stable) 


Eglin  (Wet) 


Nellis  (Dry) 


Eglin  (Wet) 


Nellis  (Dry) 


Tank 


Truck 


Tank 


Truck 


Tank 


Truck 


Tank 


AGM-65D 


AGM-65K 
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n  DOE  Screening  Design  vs 
E  OFAT 


Design  of  Experiments 
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Case|  A 

B  C 

D 

E 

F 

G 

H 

1  low  low  low 

low 

low 

hi 

hi 

hi 

2 

hi 

low 

low 

low 

low 

hi 

low 

low 

3 

low 

hi 

low 

low 

low 

low 

hi 

low 

4 

hi 

hi 

low 

low 

low 

low 

low 

hi 

5 
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low 

hi 

low 

low 

low 

low 

hi 

6 

hi 

low 

hi 
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low 

low 

hi 
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7 

low 

hi 

hi 

low 

low 

hi 

low 

low 

8 

hi 

hi 

hi 

low 

low 

hi 

hi 

hi 

9 

low 

low 

low 

hi 

low 

low 

low 

low 

10 

hi 

low 

low 

hi 

low 

low 

hi 

hi 

11 

low 

hi 

low 

hi 

low 

JSlkl 

hL 

lOWi 

hi  ■ 

fin; 

12 

hi 

hi 

low 

Jjiii 

16 


17 


18 


19 


20  hi 


low 


low 


low 


low 


hi 


low 


low 


low 


low 


low 


low 


low 


low 


low 


!i 

low 


low 


low 


low 


hi 


hi 


low 


low 


low 


low 


low  low 


low  low 
hP  hi 
hi  hi 


low  lo\ 


In  32  samples,  with  this 
fractionai  factorial  ((l/16th 
fraction  of  resoiution  IV),  we 
can  estimate  the  main  effects 
of  9  variables.  Can  collapse 
across  one  or  more  variables 
to  estimate  spread.  If  4  of  9 
factors  are  not  active,  we 
have  aii  combinations  in  any 
remaining  5  variables. 


DOE  uses  29%  of 
OFAT  samples 


Case 

A 

B 

c 

D 

E 

F 

G 

H 

J 

1 

iow 

iow 

iow 

iow 

low 

low 

low 

low 

low 

2 

hi 

iow 

iow 

iow 

low 
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low 

low 

low 

3 

best 

hi 

iow 

iow 

low 

low 

low 

low 

low 

4 

best 

hi 

low 

low 

low 

low 

low 

low 

5 

best 

hi 

low 

low 

low 

low 

low 

6 

best 

hi 

low 

low 

low 

low 

7 

an 
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hi 

low 

low 

low 

^Al 

rD< 

esi! 

best 

hi 

low 

low 

9  0 

1  ^ 

best 

hi 

low 

10 

best 

hi 

11 

best 

Y1 

. .  .1 

Yig/\Avg 

We  ’re  going  to  run 
10  because  we  ’re 
10-guys  and  that’s 
what  we  do. 
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DOE  Screening  Design  vs. 


|i  Scenario  Testing 

¥ 

¥ 


7 

8 

9 

10 


¥ 

¥ 

¥ 
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Case 

0 

T 

N 

1 

low 

low 

low/ 

2 

hi 

low^ 

3 

low 

hi 

loV 

low 

4 

hi 

hi 

low 

low 

5 

low 

low 

hi 

low, 

1119 

low 

low 


icl 


u: 


u; 


16  total 
events 

I  Vi/  V  V  I  Vi/  V  ¥ 


hi  low  low 


low 


u; 


Event 


11 

low 

hi 

low 

12 

hi 

hi 

low 

13 

low 

low 

hi 

14 

hi 

low 

hi 

15 

low 

hi 

hi 

16 

hi 

hi 

hi 

10 
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Jone: 


Jones 


In  16  events,  with  this  fractional 
factorial  matrix  ((1/2  fraction  of 
resolution  V),  testers  can  estimate 
the  effects  of  each  test  condition. 
Can  collapse  across  one  or  more 
variables  to  estimate  variance.  If  1 
of  5  conditions  don't  affect  the 
MOP,  matrix  has  all  combinations  of 
the  remaining  4  variables. 


DOE  uses  20%  of 
"Best  Guess" 
samples 
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2  2145 


2145 

2145 


Jones  2 


2  2145 
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12 

12 

12 

34 

12 

34 

34 

12 
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B-1  Radar  Mapping  in 
One  Mission  (28^^  TS) 


Design  of  Experiments 
#  - - ^  ^ 


Problem:  Characterize  B-1 
radar  coordinate  accuracy  for 
variety  of  operational  setups. 


Design: 

Response:  absolute  error 

Conditions:  angle,  side  of 
nose,  tail  number,  target, 
and  range  to  target. 

4  replicates. 


Result:  Similar 
accuracy  across  scan 
volume,  target  type,  tail 
number. 


Angular  Error  in  Target 
Coordinates  -  B-1B 
Radar  Mapping 


1. 5 


.1  ■ 


D) 

C 

< 


■1.5  ■ 


■2.D 


-0  15 

miles 
30 

miles 


Angle 

Left  side 


Angle 

Right  side 


Result:  Single  two-aircraft  mission  answered 
accuracy  questions  raised  by  7  previous 
missions  using  conventional  test  methods. 
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other  Recent  Project 
highlights  ... 


Design  of  Ex|^riments 


□  2X  increase  in  Predator  Hellfire  firing 
envelope  in  a  week's  testing 

□  50%  reduction  in  expenditures  and  runs 
required  for  B-1  Block  E  upgrade  test  with 
increased  data  fidelity 

□  45%  reduction  in  sorties  and  helped  define 
test  objectives  and  procedures  for  $325K 
HTS  verification 

□  33%  reduction  in  bombs  required  during 
evaluation  of  4  weapon  types  off  10  B-1 
sorties 

□  Uncovered  fundamental  TTPs  in  multi¬ 
platform  E-CAS  TD&E  in  40  sorties 

□  25%  reduction  in  resources  using  screening 
method  for  Chemical  Detector  test 

□  $36K  in  lab  costs  saved  for  $144K 
Expeditionary-Deployable  Oxygen 
Concentrator  System  (EDOCS)  test 

□  133  passes  to  62  in  B-1  PFS  4.2  ECM 
systems  upgrade 

We  have  more  than  150  DOE  test  projects  across 
full  range  of  ACC  systems 
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Why  DOE  —  Postscript 


q; 

Ui 

"O 

OJ 

§ 

o 

c 

U) 

U) 

Q) 

u 

o 


Perfect  Knowledge  (God) 

f 


.?.y  rr.?.  []?: . 


We  use  DOE  to  interrogate  the  process  and 
improve  our  knowledge  of  how  our  process 
works.  The  goal  is  a  systematic  method  to 
efficiently  and  unambiguously  improve  our 
outcomes. 

Compared  to  any  other  systematic  method, 
DOE  designs: 

-Yield  Better  process  understanding 

-Can  be  planned  and  analyzed  Faster 

-Are  Cheaper,  using  only  20-80%  of 
usual  resources 
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Design  of  Experiments 
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Least  You  Should  Recall... 


Three  methods  (or  strategies)  of  test  are 
commonly  used  -  OFAT,  scenario,  and 
intuition 

Intuition  is  important,  but  not  sufficient 
as  our  sole  test  strategy 

OFAT,  by  its  nature,  cannot  detect 
interactions  -  and  interactions  are 
common 

The  OFAT  search  vector  leaves  large 
areas  unexplored 

Two  variables  interact  when  the  setting 
of  one  variable  changes  the  effect  on  the 
response  of  the  second  variable 

Scenario  designs  contain  complex 
confounding  patterns  making  cause-and- 
effect  reasoning  difficult 

Scenario  designs  afford  spotty  coverage 
of  the  total  factor  space 

DOE  designs  efficiently  and  effectively 
tell  us  how  our  operations  work 


AF  T&.E  Days  -  Dec  05 


1-64 


Session  3  -  Four  Steps  to 
Design 


□  DOE  Spiral  2  -  SAM-in-a-Box  Part  Deux 

■  Decomposing  the  PROCESS 

■  Developing  the  test  PLAN 

■  PRODUCE  the  observations 

■  PONDER  what  we  have  learned 

■  Loop  ...  and  iterate 


¥ 

i 

i 

i 
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Break  -  Spiral  2  Back  to  the 
SAM-in-a-Box 
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□  We'll  do  a  simplified  multi-factor 
design  and  simplified  ANOVA 
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p  News  Flash!  US  Company 
p  Indicted  for  Iraq  Arms  Sales! 
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□  The  SAM-in-a-Box 
is  a  potent  threat! 

□  Sold  to  Saddam  by 
Stat-a-Pult  Inc. 

□  We  must  conduct 
an  exploitation 
HWIL  simulation 
today 

□  MOP  -  Launch 
Range 

□  Conditions  - 
projectile, 
propellant,  Tgt 
RCS  &  velocity, 
ECM  ...  etc 
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Tests 


—How  We  Design 
4  Stages 


Design  of  Experiments 

- - #  — ffi 

I  ?  -  ^ 

4j  ' 


I  Project  Description  and 
Decomposition 

II  Pian  the  Test  Matrix 

III  Produce  the  Observations 

IV  Ponder  the  Resuits 


We’ll  stroll  quickly  through 

9 

cxRinpics  01  6aCJi  siep  lor  iviavericK 

Problem 
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stage  I  Samples  of  Various 
Test  Objectives 
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□ 


□ 


Compare  system  to  a  fixed  standard 
(ORD,  specification,  demo  goal) 

Compare  system  to  older  version  (OFP  or 
software/hardware  upgrade,  new  mission 
data,  etc.) 

Characterize  system  performance  (test 
first,  then  evaluate  later) 

Optimize  the  performance  of  the  system 
under  test  (maximize  or  minimize) 

Make  system  most  robust  to 
environmental  conditions  (weather,  threat 
actions,  countermeasures,  operator 
experience) 

Minimize  the  variability  of  system 
performance 

Troubleshoot  faulty  system  performance 
(false  alarms) 
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Two  Tools  to  Turn  Eng  /  Ops 
|j  Art  into  Science 


w 


Measurements 

Manpower 

Materials 

Cause-Effect 
(CNX) 
Diagram 


Machines 


T- 

i 

i 
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Cause  Labels: 

C  “  Constant 
N  —  Noise  no  control 
X  —  Control  Variable 
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Tool  1  -Process  Flow  for 
I  Table  of  RESPONSES  (MOPs) 
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Tool  2  -  Fishbone  Table  of 
TEST  CONDITIONS 


Design  of  Experiments 
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iji/leasuremenh 

Manpower 

Materials 

Fishbone  or 
CNX 
Diagram 


Response 

to 

Effect 


Milieu 
(Environment) 


Methods 


Machines 


Table  of  Test  Conditions 


N 

Variable 

Units 

Range 

Priority 

Ekp.  Control 

Design  Range 
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Cause  and  Effect  Diagram 
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BTW  When  Time  is  Short  - 


p  Tool  3  Black  Box  Diagram 

P 


Process  Flow  Chart 
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Process  Black  Box  (gozinta-gozouta) 


M 


Process 


Inputs  (Test  Conditions) 


Outputs  (MOPs) 
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Design  Algorithm  for 
Full  Factorial 
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Case 

Stop 

Tgt 

Ball 

PullBack 

1 

1 

F-16 

Red 

135 

2 

1 

F-16 

Red 

180 

3 

1 

F-16 

Yellow 

4 

1 

F-16 

Yellow 

5 

1 

F-117 

6 

1 

F-117 

7 

1 

F-117 

8 

1 

F-117 

9 

3 

10 

3 

000 

000 

23 

5 

24 

5 

How  many  Variables  (A,B,C  ...  K)? 

How  many  settings  (levels)  for  each  Variable  (a,  b, 
c,  ...  k)? 

How  many  runs  total? 

■  Exannple:  4  vars:  1  at  3,  3  at  2  levels 

■  Runs  (N)  =  3*2*2*2  =  3  *  2^  =24  runs  (test  cells) 

Construct  Design  -- 

■  Var  A  set  at  1  for  N/a,  level  2,  etc. 

■  With  A  at  1,  set  B  at  1  for  N/a/b,  at  2  for  N/a/b... 

■  Continue...  Show  on  Board 


AF  T&.E  Days  -  Dec  05 


1-75 


Design  of  Experiments 


i 

i 

i 

P 

P 
P 
P 
P 
P 
P 
P 
f 

M 


I 
P 
P 
P 

I 
P 

P 


PLAN:  Create  Design 


□ 

□ 

□ 

□ 

□ 

□ 


Let  each  table  take  20  minutes  to  formulate  a 
design.  We  will  discuss  your  results  as  a  group. 

Consider  the  test  event  under  consideration 

Use  only  three  variables  at  any  number  of  levels 
each  —  instructor  will  supply  them. 

Use  instructor-supplied  measurement  as  your  MOE 
How  many  total  runs  for  one  replicate? 

Justify  your  design 
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Formulation  Sample  — 
Factorials  (Solution) 
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Case 

Ball 

Tgt 

PullBack 

1  -1  -1  -1 

1 

2 

-1 

-1 

1 

1 

3 

-1 

1 

-1 

1 

4 

-1 

1 

1 

1 

5 

1 

-1 

-1 

1 

6 

1 

-1 

1 

1 

7 

1 

1 

-1 

1 

8 

1 

1 

1 

K' 


+ 


Ball 


Pullback 


Note  the  geometric  shape  you  built 

All  variables  are  orthogonal  (right  angles) 

General  algorithm  to  build  —  vary  A  for  half  of  runs, 
B  for  half  of  A  and  C  for  half  of  B  ... 
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Some  Properties  of  Factorial 


I  Designs 
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□  Two  level  factorials  are  powerful  for  exploring 

□  Form  the  basis  for  many  more  elegant  designs 

□  Can  be  augmented  to  fit  more  complicated 


models 


Adding  Center  Points 
Measures  Curvature 
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Flexibility:  DOE  Designs  can 
be  Simple  or  Complex 


Altitude 


Low 


High 


Level 


SLD 


HARB 


Dive  Angle 


-20 


LAT 


Dive  HARB 


+30 


LOFT 


NA 


Unguided 
bomb  matrix 
from  F-15E 
Suite  4E+ 
FDE 


□  Lack  of  understanding  of  this  point  leads  to 
statements  like  : 


"DOE  is  OK  for 


but  cannot  do 


// 


DOE  Factorial  Design  with  Ctr  Points  --  Velocity  Gate  Pulloff 


—I  < 
LU  ^ 
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VGPO  Jamming 
technique 
developed  in 
Project  Six  Pack 
MDO 


18  19  20  21  22  23  24  26  26  18  19  20  21  22  23  24  26  26  18  19  20  21  22  23  24  26  26 

BEGIN:  1  BEGIN:  2  BEGIN:  4 

RCS 
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Larger  Factorial:  4  Factor 
Design 


I  Design 
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Interactions 
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Effects  and  Interactions  in 
Standard  Order 
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Standard  Order  (adding  factors+interactions)  helps 
keep  terms  straight: 
Y=M+A+B+AB+C+AC+BC+ABC+.... 
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Recall  —  we  seek  to  balance 
our  chance  of  errors 
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Putting  them 
together,  we  see 
we  can  trade  one 
error  for  the  other 
(a  for  p) 

We  can  also 
increase  sample 
size  to  decrease 
our  risks  in 
testing 

These  statements 
are  not  opinion  - 
they  are 

mathematical  fact 

and  an 
inescapable 
challenge  in 
testing 


SAM-in-a-Box  Good  =  Buy 
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Getting  it  right:  Confidence In  stating  results;  Power 

to  find  small  differences 


Replicating  a  Design  - 
I  Simple  rules  for  sample  size 
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Totai  Unique  Cases  in  Factoriai  Matrix 

Percent 

confidence  that 
identified  effect 
exists--  (1-a) 

a 

Percent 

chance  of 
finding  true 
effects  (1-p) 
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Samples  Required  Per  Case 
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Illustration:  3x2x2  matrix  gives  12 
cases.  With  a  =  p  =.05,  n  (samples 
required  per  case)  = _ 

WARNING:  Only  applicable  to 
Factorial  Designs  analyzed  with 
these  methods! ' 

Source:  Appendix  M-2,  Understanding 
Industrial  Designed  Experiments,  Schmidt 
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stage  III  PRODUCE 
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Randomized  Observations 
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□ 


Randomizing  runs  protects  from 
unknown  background  changes 
within  an  experimental  period  (due 
to  Fisher) 
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PRODUCE  observations 
in  Blocks 


Design  of  Experiments 

I  €  J 

1.1 


O  large  target 
•  small  target 


□  Blocking  designs  protects  from 
unknown  background  changes 
between  experimental  periods 
(also  due  to  Fisher) 
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p  Simple  rules  for 
p  randomization  and  blocking 
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□  Randomize  each  set  of  runs  within  a 
single  test  block,  phase  or  mission 

□  Run  one  replicate  of  the  design  for 
each  block  or  mission  (produces  a 
"block") 

□  Or  —  run  a  similar  set  of  calibration 
runs  for  each  mission  —  4  to  8  trials 
and  compare  among  blocks  to  detect 
different  performance 
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Full  Factorial  Designs 


Review  your  cause-and-effect  diagram  for  the  factors  affecting 
variability  of  the  statapult.  Based  on  the  cause-and-effect,  develop 
standard  operating  procedures  to  control  variability. 

For  this  experiment,  we  will  test  3  factors  of  interest:  1 )  stop  position, 
2)  launch  angle,  3)  tension,  and  4)  ball  type.  Each  factor  will  have  two 
levels  according  to  the  following  table.  Be  sure  to  hold  all  the  other 
factors  constant,  if  possible. 
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Li  RpII 

Type  1 

Type  2 

M  Odii 

Use  a  2^  factorial  design  approach  with  1  replicate.  Develop  a  design 
matrix  using  the  software  to  determine  the  random  order  of  the  runs. 


Perform  the  experiment  and  collect  the  data.  Make  observations 
concerning  held  constant  and  nuisance  factors  during  the  experiment  for 
future  use. 


5.  Analyze  your  data.  Determine  the  significant  effects.  Develop  a 
predictive  model  of  launch  distance.  Comment  on  your  findings  below. 

6.  Run  confirmation  runs  using  distances  provided  by  the  instructor. 
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Stage  IV  -  PONDER 
Results  (Simple  Analysis) 
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(1)  =  95  -(180) 


ab=50 


+  (135)  a=60 


PullBack 
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□ 

□ 

□ 


How  do  Pullback  &  Stop  affect  Rng? 
Effect  (Pullback)  = 

Effect  (Stop)  = 

Effect  (Pullback  X  Stop)  = 


Designed  Experiment  (n). 
Purposeful  control  of  the 
inputs  (factors)  in  such  a 
way  as  to  deduce  their 
relationships  (if  any)  with 
the  output  (responses). 


Factorial  Effect 
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Case 
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Why  this  math  model  works 
from  Algebra  II 


y  =  y-\- 


2 


2 


2 


Where  does  equation  come  from? 

1.  Orthogonal  design 

2.  Orthogonal  coding 

3.  2  levels  for  each  factor. 

Simple  y=mx+b  from  Algebra  II 


y 


Avg-  y 


-1 


Avg+ 


+1 


A 


Prediction  equation  for  one  factor  (A) 

We  can  add  other  factors  because  our  designs  are 
orthogonal  (predictors  are  independent  of  each 
other) 
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Statistics  needed  with  more 
complex  problems  ...  Where  is 
noise  floor? 


Pareto  Chart  of  Standardized  Effects;  Variable:  MSL_MD 


2**(4-0)  design;  MS  Residual=1 05910. 
DV:  MSL  MD 


p=.05 


Effect  Estimate  (Absolute  Value) 


Actual  Statistica  Pareto  chart  of  recent  test  of  an 
ECM  technique  development  test.  Four 
factors,  each  at  two  levels  gives  16 
combinations  and  15  possible  main  effects 
and  interactions. 

Red  line  drawn  with  chance  of  a  false  positive  (a 
error)  equal  to  5%. 
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Concept  of  an  External 
Reference  Distribution 
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□  Suppose  I  am  a  covert  supermodel  weigher.  I 
travel  the  world  from  Milan  to  New  York  to 
clandestinely  weigh  supermodels  in  groups  of  10 

□  Someone  offers  me  a  data  set  claiming  it's 
supermodel  weight  data.  X50  =  134  lbs,  n  =  10 

□  Examining  my  reference  collection,  I  observe  that 
only  1.25%  of  my  487  samples  equal  or  exceed 
134  lbs 

□  Accordingly,  I  reject  this  set  with  a  1.25%  change 
of  being  mistaken 

□  I  find  likelihood  (a)  that  I  see  a  median  of  134  lbs 
given  these  are  supermodel  weights,  is  low  so  I 
reject  equality 


Low  profile  electronic 
floor  scale 
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There  are  16  terms  in  our 
model  -  are  all  active? 
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Effects  and  Interactions  i 
Standard  Order 
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□  standard  Order  (adding  factors+interactions)  helps 
keep  terms  straight: 

^  Y=M+A+B+AB+C+AC+BC+ABC+-. _ 

M  AF  T&E  Days  -  Dec  05  1-91 


I 

i 

i 


ANOVA  uses  the  F  (for  Fisher) 
Reference  Distribution 
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F  Reference  Distribi  tion 


0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0 

□  In  the  Null  world,  F  ~  1.0 

□  Larger  values  of  F  lead  to  concluding 
differences  exist 
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Break  ...  weYe  limited  to  8 
!  shots/block  -  which  8  to  run? 
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Blocks  -  giving  up  some  info 
to  measure  mission  effect 
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With  four  factors  -  split  16  runs  into  two 
missions 

□  Step  1.  Build  a  full  factorial  (16  runs)  in  all 
4  vars 

□  Step  2.  Construct  high  order  interaction 
column  (ABCD) 

□  Step  4.  Confound  (alias)  ABCD  with 
missions  (blocks) 


; 

Var» 

A 

B 

C 

D 

ABCD 

1 

Case 

Block 

Proj¬ 

ectile 

Pull- 

Back 

Stop¬ 

per 

Fulcr 

um 

1 

"12 

1 

1 

1 

-1 

1 

-1 

1 

1 

1 

1 

-1 

1 

1 

-1 

1 

1 

-1 

1 

1 

1 

-1 

1 

1 

1 

-1 

-1 

-1 

-1 

1 

1 

-1 

1 

-1 

-1 

-1 

1 

1 

-1 

-1 

1 

-1 

-1 

1 

1 

1 

1 

1 

-1 

-1 

1 

1 

-1 

-1 

-1 

1 

-1 

1 

2 

-1 

-1 

-1 

-1 

1 

1 

'"lO 

2 

1 

-1 

-1 

1 
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2 

-1 

1 

-1 

1 

1 

1 

2 

-1 

-1 

1 

1 

1 

1 

2 

1 

1 

1 

1 

1 

1 

2 

1 

1 

-1 

-1 

1 

1 

2 

1 

-1 

1 

-1 

1 

1 

2 

-1 

1 

1 

-1 
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I  What  can  we  learn  from  the  second 
I  Block?  -  fraction  of  a  full  factorial 
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Seldom  do 
this  small 
half  fraction 
since  two- 
ways  are 
confounded 
with  each 
other 


P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 
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Given  the  number  of  factors  k=4 
Full  fraction  has  2^=16  runs; 

Half  fraction  has  2^'^  =  2^  =  8  runs 

I  Step  1.  Build  a  full  factorial  (8  runs)  in  first 
3  vars 

I  Step  2.  Alias  (perfectly  confound)  fourth 
factor  with  highest  interaction  (three  way) 
in  Step  1. 

□  Step  3.  Determine  Aliasing  (confounding 
pattern)  for  all  effects. 


Rules  for 

Confound! 

Patterns: 

I*A=A 


Var» 

A 

B 

c 

D=ABC 

Case 

Block 

Proj 

Pull 

stop 

Pule 

"1 

2 

-1 

-1 

-1 

-1 

■'13 

2 

-1 

-1 

1 

1 

"11 

2 

-1 

1 

-1 

1 

2 

-1 

1 

1 

-1 

"10 

2 

1 

-1 

-1 

1 

"6 

2 

1 

-1 

1 

-1 

2 

1 

1 

-1 

-1 

ig 

2=1 

2 

1 

Using  this  group  1 

Algebra— 

D=ABC 

I=ABCD 

r 

P 
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Five  Variable  Half  Fraction 


Case 

A 

B 

C 

D 

E=ABCD 

1 

-1 

-1 

-1 

-1 

1 

2 

-1 

-1 

-1 

1 

-1 

3 

-1 

-1 

1 

-1 

-1 

4 

-1 

-1 

1 

1 

1 

5 

-1 

1 

-1 

-1 

-1 

1 

6 

-1 

1 

-1 

1 

1 

1 

7 

-1 

1 

1 

-1 

1 

1 

1 

8 

-1 

1 

1 

1 

-1 

1 

1 

9 

1 

-1 

-1 

-1 

-1 

1 

10 

1 

-1 

-1 

1 

1 

1 

11 

1 

-1 

1 

-1 

1 

1 

12 

1 

-1 

1 

1 

-1 

1 

13 

1 

1 

-1 

-1 

1 

1 

14 

1 

1 

-1 

1 

-1 

1 

1 

15 

1 

1 

1 

-1 

-1 

1 

16 

1 

1 

1 

1 

1 

Step  1.  Build  a  full  factorial  (2^'^  runs)  in  first  k-1 
variables 

Step  2.  Alias  (perfectly  confound)  kth  factor  with 
highest  interaction  (2^'^  -  way)  in  Step  1. 

Step  3.  Determine  Aliasing  (confounding  pattern)  for 
all  effects. 


Using  group  Algebra- 
E=ABCD 
I=ABCDE 
other  aliases? 
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The  Least  You  Should  Recall... 


□ 


□ 

□ 


□ 


Four  Steps  to  DOE  Include: 

■  Process  -  decompose  with  flow  & 
fishbone 

■  Plan  -  build  a  factorial/crossed  design 

■  Produce  -  buy  insurance  with 
randomization  &  blocking 

■  Ponder  -  use  averages  and  ANOVA  to 
draw  conclusions 

Do  not  neglect  power  of  simple  tools 

Factorials  solve  the  deep  &  broad 

problem 

Unknown-unknowns  lurk 


□  Simple  analysis  uses  geometry  & 
slopes 

□  More  complex  blocks  lead  to  idea  of 
fractional  factorials 
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Session  4  Fractional 
Factorials  and  Advanced 
Topics 


□  Overview  of  DOE  Designs 

□  Fractional  Factorials 

□  Dangerous  Designs 

■  Plackett  Burman 

■  Taguchi  Orthogonal  Arrays 

■  Optimal  Designs 

□  Variations  on  Factorials 

■  Random  effects,  Nested,  Split  Plot, 
ANCOVA 

□  Mixed  Models 

□  The  3-level  model 
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Blocking  (again) 
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Q 

E 

o 

O 

"c 

0 

E 

0 

0 


ab 


ac 


be 


abc 


BIk  1 


(1) 

b 

ab 

c 

ac 

be 

abc 


Factorial  Effect 


BIk  2 


(1) 

b 

ab 

c 

ac 

be 

abc 


B 


BIk  3 


(1) 

b 

ab 

c 

ac 

be 

abc 


3  Reps  of  a 
2^  in  3 
blocks 


□  Block  -  homogeneous  experimental  unit  that 

■  Potentially  affects  the  response  and 

■  Restricts  Randomization 

□  Examples:  missions,  altitudes,  pilots,  aircraft, 
threat  serial  numbers,  .... 

□  Simplest  use  of  blocks  is  building  up  reps 

□  Example  --  2^  design  in  a  12  pass  mission 

I  - 
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Design  of  Experiments 

#  - - ♦  Sk-, 


Let's  Stroll  Through  Designs 


Design  &  Analysis  of  Ekp 


Statistica  Designs 


Quick  Advanced 


2’‘’‘(K-p)  standard  designs  (Box,  Hunter,  &  Hunter) 


2-level  screening  (Plackett-Burman)  designs 
2’‘’‘(K-p)  max  unconfounded  or  min  aberration  designs 


im 

A 


3’‘’‘(K-p)  and  Box-Behnken  designs 

Mixed  2  and  3  level  designs 

Central  composite,  non-factorial,  surface  designs 

Latin  squares,  Greco-Latin  squares 

T  aguchi  robust  design  experiments  (orthogonal  arrays) 

Mixture  designs  and  triangular  surfaces 


Designs  for  constrained  surfaces  and  mixtures 
|d|  D-  and  A-  (T-)  optimal  algorithmic  designs 


□  K 


] 


Cancel 


Options  ^  I 

For  full-factorial 
designs,  hierarchically 
nested  models  or 
designs  with 
unbalanced  nesting, 
and  miKed-model 
(random  effect) 
designs,  see  also 
Variance  Components 
and  GLM. 

1?^  Open  Data  | 


SELECT  ^ 

CHSES  ^ 


I  ^ al 


□  DE  6  Offers  these  designs 

■  2  Level  factorials 

■  Irregular  Fractions 

■  General  factorials  (including  3k  designs) 

■  D-Optimal  designs 

■  Plackett  Burman  screening  designs 

■  Taguchi  Orthogonal  Arrays 

■  Plus: 

□  (Mixture  designs) 

□  (Designs  for  constrained  regions) 
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Blocking  Replicates  — 
ANOVA  Table 


□ 


□ 


□ 


Source 

df 

SSquares 

MSquare 

F  Statistic 

P  value 

( 

Blocks 

bl-1=2 

SSBIocks 

i 

A  (ECM) 

1 

SSA 

MSA 

FA 

0.000 

4 

B  (Altitude) 

1 

SSB 

MSB 

FB 

0.000 

1 

C  (Offset) 

1 

SSC 

MSC 

FC 

0.000 

( 

AB 

1 

SSAB 

MSAB 

FAB 

0.000 

i 

AC 

1 

SSAC 

MSAC 

FAC 

0.000 

BC 

1 

SSBC 

MSBC 

FBC 

0.000 

i 

ABC 

1 

SSABC 

MSABC 

FABC 

( 

Error 

14 

SSE 

MSE 

1 

Total 

N-1=23 

There  are  b-1  degrees  of  freedom 
associated  with  the  blocks  and  b 
blocks. 

Note  no  block-treatment 
interaction 

Note  no  block  F  statistic 


AF  T&.E  Days  -  Dec  05 


I-lOl 


¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

P 

¥ 

I 

¥ 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 


22  Factorial  in  2  Blocks 


Factorial  Effect 


o 

E 

o 

O 

C 

CD 

E 


I 

A 

B 

AB 

(1) 

+ 

- 

- 

+ 

a 

+ 

+ 

- 

- 

b 

+ 

- 

+ 

- 

ab 

+ 

+ 

+ 

+ 

BIk  1  BIk  2 


(1) 

ab 


a 

b 


A=  ab+a-(l)-b 

B=ab+b-(l)-a 

AB=ab+(l)-a-b 

Blocks=? 


□  Confound  (or  intentionally  intertwine) 
AB  with  Blocks 

□  Note  that  A  and  B  are  unaffected  by 
blocks  --  there  are  one  plus  and  minus 
in  each  block  and  the  block  effect  will 
cancel  out 


AF  T&.E  Days  -  Dec  05 


1-102 


Design  of  Experiments 


¥ 

¥ 

¥ 

¥ 

¥ 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 

P 


2^  in  Two  Blocks 


Factorial  Effect 


I 

A 

B 

AB 

C 

AC 

BC 

ABC 

(1) 
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+ 

+ 
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c 
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- 
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+ 
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- 

- 

+ 

ac 

+ 

+ 

- 

- 

+ 

+ 

- 

- 

be 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

abc 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

O 

Q 

E 

o 

O 


CD 


BIk  1 


BIk  2 


(1) 

ab 

ac 

be 


a 

b 

c 

abc 


□ 


Confound  ABC  with  blocks 

■  Method  1  -  choose  +/-  signs  in  ABC 
contrast 

■  Method  2  -  let  stats  program  do  it 

■  There  are  two  other  methods  ... 
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Blocking  Generators 

Design  of  Exper 

1 

■  me 

-M  - 

iments 

■r 
1  J 
#  ' 

w 

Design:  2* 

*^(5-0)  design  in  4  Blocks 

1 

123 

Statistica  Generators 

ll 

I 

2 

345 

ABC 

CDE 

Block 
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B 

c 

D 

E 

ABC 
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L2 

1 
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I  □  ABC*CDE=ABC2DE=ABDE 
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Intro  to  Fractional 
Factorials 


Design:  2**(4-0)  desig 

n  (seatwork  lO.sta) 

A 

B 

C 

D 

Block 

(Cont.) 

(Cont.) 

(Cont.) 

(Cont.) 

8 

1 

1 

1 

1 

-1 

I 

1 

1 

-1 

-1 

-1 

-1 

10 

1 

1 

-1 

-1 

1 

15 

1 

-1 

1 

1 

1 

4 

2 

1 

1 

-1 

-1 

II 

11 

2 

-1 

1 

-1 

1 

5 
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-1 

-1 

1 

-1 

14 

2 

1 

-1 

1 

1 

6 

3 

1 

-1 

1 

-1 

III 

3 

3 

-1 

1 

-1 

-1 

12 

3 

1 

1 

-1 

1 

13 

3 

-1 

-1 

1 

1 

7 

4 

-1 

1 

1 

-1 

IV 

2 

4 

1 

-1 

-1 

-1 

9 

4 

-1 

-1 

-1 

1 

16 

4 

1 

1 

1 

1 

Guiding  Principles  Make  Fractions  Attractive 


□  Sparcity  of  Effects:  There  are  63  df  in  2® 
design  —  less  than  10%  of  effects  and 
interactions  are  usually  "significant" 

□  Projection  Property:  Most  fractional  factorials 
project  into  a  (possibly)  replicated  full 
factorial  in  fewer  variables 

□  Sequential  Experimentation:  It  is  better  to 
let  facts  drive  you  from  design  to  design 
rather  than  opinion 
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One 

Half  Fraction  2^  inTwa  Block 


Design  of  Experiments 


Factorial  Effect 


I 

A 

B 

AB 

C 

AC 

BC 

ABC 

(1) 

+ 

- 

- 

+ 

- 

+ 

+ 

- 

a 

+ 

+ 

- 

- 

- 

- 

+ 

+ 

b 

+ 

- 

+ 

- 

- 

+ 

- 

+ 

ab 

+ 

+ 

+ 

+ 

- 

- 

- 

- 

c 

+ 

- 

- 

+ 

+ 

- 

- 

+ 

ac 

+ 

+ 

- 

- 

+ 

+ 

- 

- 

be 

+ 

- 

+ 

- 

+ 

- 

+ 

- 

abc 

1/^ 

V  + 

+ 

+ 

+ 

+ 

+ 

+ 

O 

Q 

E 

o 

O 


CD 


□ 


□ 


□ 


BIk  2 


a 

b 

c 

abc 


CoPtfeand  ABC  with  Fraction  (same 
procedure  as  blocks) 

■  Method  1  —  choose  +/-  signs  in  ABC 
contrast 

■  Method  2  —  defining  contrasts 

This  is  a  2  design  --  a  half  fraction  of 
2^  ■■  four  runs  vs.  8 

ABC  is  our  generator  —  equal  (in 
principal  fraction)  to  I 
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Design  of  Experiments 


Projection  of  Fractions 


A 


□  This  half  fraction  projects  into  full 
fraction  of  2^  design  in  any  of  three 
variables 

□  In  general,  a  design  of  R,  projects  into 
possibly  replicated  factorial  of  R-1 
variables 

□  Since  max  resolution  of  half  fraction  of  2'< 
is  k,  every  2  fraction  projects  into  full 
factorial  in  any  k-1  factors,  2  reps  in  k-2, 
etc... 

□  Not  generally  true  for  higher  fractions 
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Example  Half  Fraction 


Temperature 

Content 

Treatment 

Refiner 

Length 

-1 

-1 

-1 

-1 

1.71 

1 

-1 

-1 

1 

1.86 

-1 

1 

-1 

1 

1.79 

1 

1 

-1 

-1 

1.67 

-1 

-1 

1 

1 

1.81 

1 

-1 

1 

-1 

1.25 

-1 

1 

1 

-1 

1.46 

1 

1 

1 

1 

0.85 

□  Suppose  only  one  half  fraction  of 
problem  7-15  could  be  run  2 

□  Create  design  with  DEG 

□  Run  analysis  using  LEAP  but  Verify! 

□  Check  results  with  original  and  half 
fraction  of  Replicate  2 
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One  Quarter  Fraction  of 
2^  Design  (2'^'2  ) 
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Label 

A 

B 

C 

D=AB 

E=AC 

(1) 

-1 
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-1 
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1 
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-1 

c 
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-1 
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1= 

1 
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□  Write  FF  Basic  design  in  k-2  factors 

□  Confound  (equate)  k-1,  k  factors  with  2 
appropriate  effects  (e.g.,  P  and  Q) 

□  Compute  settings  for  k,  k-1  as  effect 
products 

□  Generator  is  P,  Q  and  PQ  generalized 
interaction 
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^  1 

M  Quarter  Fraction  Example  [ 

W  1 

Quarter  Fraction  of  26  design  --  l=ABCE=BCDF=ADEF 

P 

—  Basic  Design - 

Fractional  Generators 

a 

Case 

Label 

A 

B 

C 

D 

E=ABC 

F=BCD 

1 

(1) 

-1 

-1 

-1 

-1 

-1 

-1 

2 

a 

1 

-1 

-1 

-1 

1 

-1 

1 - 1  A 

3 

b 

-1 

1 

-1 

-1 

1 

1 

H 

4 

ab 

1 

1 

-1 

-1 

-1 

1 

a 

5 

c 

-1 

-1 

1 

-1 

1 

1 

i - 

6 

ac 

1 

-1 

1 

-1 

-1 

1 

M - ' 

7 

be 

-1 

1 

1 

-1 

-1 

-1 

W= 

8 

abc 

1 

1 

1 

-1 

1 

-1 

P - ' 

9 

d 

-1 

-1 

-1 

1 

-1 

1 

- ' 

10 

ad 

1 

-1 

-1 

1 

1 

1 

i - 

11 

bd 

-1 

1 

-1 

1 

1 

-1 

M - ' 

12 

abd 

1 

1 

-1 

1 

-1 

-1 

i’ - ■ 

1 - 1  A - 

13 

cd 

-1 

-1 

1 

1 

1 

-1 

P 

14 

acd 

1 

-1 

1 

1 

-1 

-1 

a 

IM - ■ 

15 

bed 

-1 

1 

1 

1 

-1 

1 

w — 

16 

abed 

1 

1 

1 

1 

1 

1 

i 

i 

i 

i 

i 

i 

i 

i 

p 

i 

i 


□  Montgomery  example  8-4,  a  quarter  fraction 
of  2^  design:  2^-2 

□  Generate  in  Statistica  (careful  of  chosen 
generators) 

□  Objective  —  minimize  shrinkage  after  24 
hours 

□  Response  --  percent  shrinkage  (transform??) 

□  Factors  — 

■  A—  Mold  temperature  B--  Screw  speed,  C— 
hold  time,  D--Cycle  time,  E—  Gate  size,  F-- 
hold  pressure 
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Analysis  Results  —  3-ways 
assumed  negligible 


□ 

□ 

□ 


Probability  Plot;  Var.:SHRINKAG;  R-sqr=.98572 
2**(6-2)  design;  MS  Residual=47.5625 
DV:  SHRINKAG 


.99 

.95 

.85 

.75 

.65 

.55 

.45 

.35 

.25 

.15 

.05 

.01 


Standardized  Effects  (t-values) 

-  Interactions  +  -  Main  effects  and  other  effects 


A,  B  and  AB  stand  out  with  AD  as  distant 
possibility 

Rule  --  be  flexible  and  humble  but  a  little 
ruthless  with  p-values  -  you  cannot  consider 
everything! 

Get  the  Pareto  results  —  80% 


Pareto  Chart  of  Standardized  Effects;  Variable:  SHRINKAG 
2**(6-2)  design;  MS  Residual=47.5625 
DV:  SHRINKAG 
p=.05 


(2)SCREWSPD 

(1)MOLDTEMP 

1by2 

1by4 

1by5 

1by3 

(4) CYCLTIME 
(3)HOLDTIME 

1by6 

(5) GATESIZE 
(6)HOLDPRES 

2by4 

2by6 


I  -.54375 
I  -.47125 
I  .3987497 
^  -.25375 
^  .1812499 
^  .1087499 
^  .1087499 
^  -.03625 
■  -.03625 


^  -1.55875 


.023747 

J747 

^  10.33124- 

1  Sverdrup  I 

AF  T&E  Days  - 


2  4  6 

Effect  Estimate  (Absolute  Value) 


I-lll 


Design 


D=+AB 

l=ABD 


p  Res  III  Design  Issues 

P 
P 

P 
P 
P 
P 
P 
P 
P 
P 
P 


Design  of  Experiments 

I  <  ~1J 


G=+ABC 

l=ABCG 


-1 


1 


-1 


-1 


-1 


Tr 

Combo 


def 


afg 


beg 


abd 


cdg 


ace 


bcf 


abcdefg 


Time 


85.5 


75.1 


93.2 


145.4 


83.7 


77.6 


95.0 


141.8 


1^ 

I 

SS 

df 

MS 

IJJJJL 

> 

850.781 

1 

850.781 

|i  (2)B 

2945.281 

1 

2945.281 

'i  (3)C 

0.151 

1 

0.151 

|i  (4)D_AB 

1667.531 

1 

1667.531 

1  (5)E  AC 

0.151 

1 

0.151 

|i  (6)F BC 

0.781 

1 

0.781 

1  (7)G_ABC 

11.761 

1 

11.761 

i  Error 

0.000 

0 

i  Total  SS 

5476.439 

7 

Factor 


m 


(2)B 


(3)C 


(4)D  AB 


(5)E  AC 


(G)F  BC 


[7)G  ABC 


Confouriding  of  Effec 


Alias 

1 


2*4 


Alias 

2 


Alias 

3 


1*4 

1*5 

1*2 

1*3 

1*7 

1*6 


3*5 

3*6 


6V 


2*6 


3V 


2*7 


2*3 


5*7 


4*7 


5*6 


4*6 


4*5 


2*5  3*4 


M 

M 

P 

P 


□  Monty  example  8-7,  a  human  factors 
experiment  1/16^^  fraction  of  a  IP  design. 

□  What  can  we  do  with  the  aliases  of  B  &  D? 
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Res  III  Foldover  Designs 

Design  of  Experiments 

M 

Res  III  Design 

CASE 

A 

B 

C 

U=+AB 

l=ABD 

t=+AC 

l=ACE 

l-=+BC 

l=BCF 

G=+ABC 

l=ABCG 

Tr 

Combo 

Time 

1 

-1 

-1 

1 

1 

1 

-1 

def 

85.5 

2 

1 

-1 

-1 

-1 

-1 

1 

1 

afg 

75.1 

3 

-1 

1 

-1 

-1 

1 

-1 

1 

beg 

93.2 

4 

1 

1 

-1 

1 

-1 

-1 

-1 

abd 

145.4 

5 

-1 

-1 

1 

1 

-1 

-1 

1 

cdg 

83.7 

6 

1 

-1 

1 

-1 

1 

-1 

-1 

ace 

77.6 

7 

-1 

1 

1 

-1 

-1 

1 

-1 

bcf 

9 

5.0 

8 

1 

1 

1 

1 

1 

1 

1 

abcdefg 

14 

1.8 

\ 

□ 

□ 


□ 

□ 


Monty  8-7  —  Problems  with  aliases,  projectior 

Solution  -  Fold  over  by  Reversing  the  signs  in 
generators  with  odd  number  of  letters  in  their 
word 

The  odd  letters  alias  mains  with  2-ways 


Plus  Complete  Foldover  Gives  Res  IV 

CASE 

ABC 

D=-AB  E=-AC 

l=-ABD  l=-ACE 

F=-BC 

l=-BCF 

G=+ABC 

l=+ABCG 

Tr 

Combo 

T^e  I 

9 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

(1) 

71.9 

10 

1 

-1 

-1 

1 

1 

-1 

1 

adeg 

87.3 

11 

-1 

1 

-1 

1 

-1 

1 

1 

bd1g 

143.8 

12 

1 

1 

-1 

-1 

1 

1 

-1 

abef 

94.1 

13 

-1 

-1 

1 

-1 

1 

1 

1 

cefg 

73.4 

14 

1 

-1 

1 

1 

-1 

1 

-1 

acdf 

82.4 

15 

-1 

1 

1 

1 

1 

-1 

-1 

bode 

136.7 

16 

1 

1 

1 

-1 

-1 

-1 

1 

abcg 

91.3 
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Design  of  Experiments 


i 

It 


Results  of  the  Fold  Over 


n  j 


SS 

df 

MS 

F 

P 

(1)A 

8.70 

1 

8.7 

0.52 

0.60 

(2)B 

5791.21 

1 

5791.2 

344.51 

0.03 

(3)C 

12.96 

1 

13.0 

0.77 

0.54 

(4)D AB 

3451.56 

1 

3451.6 

205.33 

0.04 

(5)E AC 

0.06 

1 

0.1 

0.00 

0.96 

(6)F BC 

1.00 

1 

1.0 

0.06 

0.85 

(7)G ABC 

0.06 

1 

0.1 

0.00 

0.96 

1  by  2 

1.00 

1 

1.0 

0.06 

0.85 

1  by  3 

0.64 

1 

0.6 

0.04 

0.88 

1  by  4 

0.42 

1 

0.4 

0.03 

0.90 

1  by  5 

9.30 

1 

9.3 

0.55 

0.59 

1  by  6 

26.01 

1 

26.0 

1.55 

0.43 

1  by  7 

5.06 

1 

5.1 

0.30 

0.68 

2  by  4 

1466.89 

1 

1466.9 

87.26 

0.07 

Error 

16.81 

1 

16.8 

Total  SS 

10791.70 

15 

P 

P 

P 

P 

P 

i 

P 

P 

P 

P 

i 

P 
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□  Res  IV  design  indicates 
it's  B,  D  and  BD 
interaction 

□  B  and  D  de-aliased  with 
two  factor  interactions 


Factor 


OJA 


f4)D  AB 


f5)E  AC 


(B)F  EC 


f7)G  ABC 


1  by  2 


1  by  3 


1  by  4 


1  by  5 


1  byG 


1  by  7 


2  by  4 


Confounding  ( 


Alias 

1 


Alias 

2 


3V 

2V 

3*6 

2*6 

2*5 

2*3 

3*5 


5*6 


4*6 


5*7 


4*7 


3*4 


4*5 


6*7 
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u 


^>8^imple  Rules  for  De- 
Aliasing 


Design  of  Ex|^riments 


1.  Run  the  other  half  fraction 

2.  Augment  fraction  to  break  selected 
chains 


■  (Hidinger  Conjecture  -  2  runs  break  2  chains 

3.  Assume  away  3-way  (4-Way)  and 
higher  interactions 

4.  Subject  matter  reasoning  -  this  one, 
not  that 


5.  Collapse  across  inactive  var  & 
project 

6.  Predict  and  confirm  results 

7.  Foldover  RIII  to  RIV  (Complete 
foldover) 

8.  Foldover  on  a  letter:  RIII  ->  RV  one 
letter 


9.  Ockham's  razor  -  big  main  effects 
=  >  guilty  of  big  two  ways 
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Three  level  designs  have 
drawbacks 


From:  3**(3-0)  full  factorial  design,  1  block  ,  27  runs 


□  Sometimes  used  when  2nd  order  models 

or  curvature  are  desired 

■  Response  surface  designs  (central  composite 
designs)  superior 

■  2  level  factorial  with  center  points  ok  for 
curvature 

□  Details 

■  Each  variable  A,  B,  C,...k  has  three  levels  0,1,2 

■  Total  of  3^  cases  in  design 

■  Main  effects  each  have  2  df  vs.  1  in  2^  designs 

■  Can  be  used  to  estimate  linear  and  quadratic 
elements 
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has  k  digits  —  0,1,2  denoting  factor  levels. 
All  at  low  denoted  000 
All  high  denoted  222 
A  high,  B  med  C  low  denoted  210 
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□  New  notation  —  012  or  familiar  -101  to 
denote  levels 

□  For  32  design  --  9  runs  with  8  df 

□  2-way  interactions  (e.g.  AB)  have  (a- 
l)*(b-l)=4  df 

■  partitioned  into  LxL  LxQ  QxL  and  QxQ 

■  Possible  to  partition  in  AB  and  AB^  (I,J) 
also 
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Design  of  Experiments 


P 

¥ 

¥ 

¥ 

¥ 
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Example  3^  design 


¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 


Effect 

(1)TOOL ANG  L+Q 

SS  ^  '  “f|s 

24  Usucil  ANOVA  ^2  17 

F 

8.42 

P 

0.009 

(2)CUT SPD  L+Q 

25.33 

2 

12.67 

8.77 

0.008 

1*2 

61.33 

4 

15.33 

10.62 

0.002 

Error 

13 

9 

1.44 

Total  SS 

124 

17 

□  Tool  life  study  from  Montgomery 
example  6-5 

□  Note  expanded  effects 

□  Plot  of  two  way  interactions 
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Design:  2  3-level  factors,  1  Blocks,  18  Runs 
NOTE:  Std.Errs.  for  means  computed  from  MS  Error=  1.444444 


1 

1 

Effect 

SS  1 

1  df  1 

Sis 

F 

P 

1 

(1)TOOL ANG(L) 

Partitioned  ANOVA 

5.8 

0.040 

1 

TOOL ANG(Q) 

lU.U 

1 

16.0 

11.1 

0.009 

1 

(2)CUT SPD  (L) 

21.3 

1 

21.3 

14.8 

0.004 

1 

CUT SPD  (Q) 

4.0 

1 

4.0 

2.8 

0.130 

1 

ILby  2L 

8.0 

1 

8.0 

5.5 

0.043 

1 

ILby  2Q 

42.7 

1 

42.7 

29.5 

0.000 

1 

IQ  by  2L 

2.7 

1 

2.7 

1.8 

0.207 

1 

IQ  by  2Q 

8.0 

1 

8.0 

5.5 

0.043 

1 

Error 

13.0 

9 

1.4 

1 

Total  SS 

124.0 

17 

CUT_SPD 

-1. 

CUT_SPD 

0. 

CUT_SPD 

1. 


Design  of  Experiments 
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The  33  design 


□  For  3^  design  --  27  runs  with  26  df 

□  Main  effect  sum  of  squares  partitioned  as 
before 

□  Two  ways  partitioned  as  before 

□  Three  ways  with  (a-l)*(b-l)*(c-l)=8  df 
—  LxLxL,  LxQxL,  LxQxQ,  QxLxL,  etc... 

■  (Usually  combine  three  ways  and  higher 
into  single  interaction  term  since  these 
are  difficult  to  interpret) 

□  If  one  or  more  factors  qualitative  -- 
create  quadratic  models  for  each  of  the 
levels  of  the  qualitative  variable 
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Design  of  Experiments 
#  - - ^  ^ 
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Confounding  in  Design 


□ 


□ 

□ 

□ 

□ 


3'<so  large  that  confounding  in  blocks  or 
fractionation  often  physicaily  required 
35  is  243  runs  (almost  8x  equivalent  2^ 
design 

3'<  confounded  in  3p  (p<k)  biocks  --  3 
biocks,  9  etc 

Confound  two  df  effect  with  biocks  — 
e.g.  AB2 

Usuai  defining  contrast  method  used. 
Can  do  3'^  factoriai  in  9  blocks  as  well 
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E  Fractionating  3*^  design 


One  third  fraction  of  3^  design 


P  □  As  in  P-B  designs,  have  to  deal  with  partial 
aliasing  of  partially  quadratic  effects. 


'io  Correlations  of  Effects  (monty  6-5  and  10-1  3^2. sta) 


1  EXPERIM. 

3**C3- 

1)  fractional 

factorial  design. 

1  block  ,  9  r 

'  DESIGN 

1  Factor 

C 

Linea 

C 

Quadra 

1L*2L 

1L*2Q 

1Q*2L 

1Q*2Q 

1L*3L 

1L*3Q 

(1)A 

CL) 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

1  (2)B 

(L) 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

-.71 

1  ^ 

(Q) 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

0  .  00 

-.71 

0  .  00 

(3)C 

(L) 

1.00 

0.00 

0.00 

-.71 

-.71 

0.00 

0.00 

0.00 

’  C 

CQ) 

0  .  00 

1  .  00 

-.71 

0  .  00 

0  .  00 

.  71 

0  .  00 

0  .  00 

1  1L*2L 

0  .  00 

-.71 

1  .  00 

0  .  00 

0  .  00 

0  .  00 

.  50 

0  .  00 

1  1L*2Q 

-.71 

0  .  00 

0  .  00 

1  .  00 

0  .  00 

0  .  00 

0  .  00 

-.50 

M 

.  1Q*2L 

-.71 

0  .  00 

0  .  00 

0  .  00 

1 .  00 

0  .  00 

0  .  00 

.  50 

1Q*2Q 

0  .  00 

.  71 

0  .  00 

0  .  00 

0  .  00 

1  .  00 

.  50 

0  .  00 

1L*3L 

0.00 

0.00 

.50 

0.00 

0.00 

.50 

1.00 

0.00 

I  1L*3Q 

0.00 

0.00 

0.00 

-.50 

.  50 

0.00 

0.00 

1.00 

1  1Q*3L 

0  .  00 

0  .  00 

0  .  00 

.  50 

-.50 

0  .  00 

0  .  00 

0  .  00 

.  1Q*3Q 

0  .  00 

0  .  00 

.  50 

0  .  00 

0  .  00 

.  50 

0  .  00 

0  .  00 

1  2L*3L 

0  .  00 

0  .  00 

.  50 

0  .  00 

0  .  00 

.  50 

.  50 

0  .  00 

2L*3Q 

0  .  00 

0  .  00 

0  .  00 

.  50 

-.50 

0  .  00 

0  .  00 

-.50 

1  H 

n  nn 

i  n  np.... 

™J0L  nn 

n  nn^ 

.  n  nn  1 

1  ± 

r 
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Alias  structure  is  complex 

Running  one  of  the  three  blocks  of  a  3^"' 

™  A=A(AB2C2)=A2B2C2=ABC 
A=A(AB2C2)2=A3B4C‘'=BC 
B=B(AB2C2)=AB3C2=AC2 
B=A(B2C2)2=A2B5C4=ABC2 
C=C(AB2C2)=AB2C3=AB2 
C=C(AB2C2)=A2B''C5=AB2C 
AB=AB(AB2C2)=A2B3C2=AC 
AB=AB(AB2C2)2=A3B5C''=BC2 
or 

la  =  A+BC+ABC 
lb  =  B+AC2+ABC2 
Ic  =  C+AB2+AB2C 
lab  =  AB+AC+BC2 


Design  of  Experiments 
#  - - ^  ^ 


■-t  ^  r 


□  Since  main  effects  are  aiiased  with  2 
factor  —  RIII 

□  If  any  two  factor  interactions  are  large  -- 
difficult  to  isolate  with  these  designs 

□  one-ninth  designs  more  complex. 

□  No  simple  augmentation  schemes 
(foldover) 

□  Monty:  designs  are  solution  looking 

for  problem  " 
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I  Designs  at  two  and  four  level 


Relatively  simple  — 
treat  four  level  as  2x2 
level  factors 

X=  1,2, 3, 4 

A  maj 

Dping 

;  between  ] 

K  and  AxE 

B  =  -l 

B  =  1 

A  =  -l 

X=1 

X  =  3 

A=  1 

X  =  2 

X  =  4 

A 

B 

=X 

C 

D 

AB 

AC 

BC 

ABC 

AD 

BD 

ABD 

CD 

ACD 

BCD 

ABCD 

■1 

■1 

1 

-1 

-1 

1 

1 

1 

-1 

1 

1 

-1 

1 

-1 

-1 

1 

1 

■1 

2 

-1 

-1 

-1 

-1 

1 

1 

-1 

1 

1 

1 

1 

-1 

-1 

■1 

1 

3 

full  design 

-1 

1 

1 

-1 

1 

1 

-1 

1 

-1 

1 

1 

4 
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-1 

-1 
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1 
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-1 

-1 

1 
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-1 
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-1 
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-1 

1 
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-1 

■1 

1 

3 

1 

1 

-1 

-1 

1 

-1 

-1 

1 

-1 

1 

-1 

1 

-1 

1 

1 

4 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Run 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 


¥ 


1 

1 

Source 

SSQ 

df 

MS 

1 

X 

SSA+SSB+SSAB 

3 

MSX 

1 

C 

SSC 

1 

MSC 

1 

D 

SSD 

1 

MSD 

1 

1 

CD 

SSCD 

1 

MSCD 

1 

XC 

SSAC+SSBC+SSABC 

3 

MSXC 

1 

SSXD 

SSAD+SSBD+SSABD 

3 

MSXD 

1 

SSXCD 

SSACD+SSBCD+SSABCD 

3 

MSXCD 

1 

1 

Error 

SSE 

0 

MSE 

1 

Total 

SST 

15 
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Objective  of  these  last  topics 


□ 


□ 


□ 


□ 


There  are  a  number  of  other  DOE 
Topics  we  have  not  addressed 

These  other  topics  are 
occasionally  useful 

You  should  be  aware  of  them  and 
know  where  to  find  them 

If  needed  -  two  alternatives: 

■  Teach  yourself  (already  have 
foundations) 

■  Pete  V's  Rule  1  of  Analysis: 

Seek  professional  help! 


Mainline  Classes  of  Designs 


Design  of  Eb^periments 

;  'r.  ]l 


¥ 

¥ 

¥ 


¥ 

i 

i 

¥ 

¥ 

¥ 

¥ 

i 

¥ 

¥ 

¥ 
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ANCOVA  -  Analysis  of 
Covariance 


Design  of  Experiments 
#  - - ^  ^ 


Scatterplot  (Monty  4-4  ANCOVA.STA  4v*15c) 


□ 


□ 

□ 

□ 


□ 


FACTORIAL  of  all  Effects;  design:  (monty  4-4  ancova.sta) 

1-MACHINE 

df 

MS 

df 

MS 

Effect 

Effect 

Error 

Error 

F 

p-level 

1 

2 

34.86666 

12 

25.5 

1.36732 

0.291776 

o  MACHINE:  1 
□  MACHINE:  2 


X_DIAM 

ANCOVA  of  all  Effect: 

3;  design:  (i 

monty  4-4  c 

ancova.sta) 

1 

1 -MACH  IN 

E 

df  / 

l0js  "X 

df 

MS 

( 

U 1  / 

Effect  / 

IVIO  \ 

Effect 

trror 

1 VI  o 

Error 

F 

p-level 

1 

0.874629 

)  11 

8.870948 

0.098595 

0.906901 

Used  when  some  X  is  random, 
uncontrollable,  but  measurable  and 
systematically  causes  Y  to  vary  with  it. 

Examples  --  Cj  with  Mach  and  MD  orTE 
with  Range 

Specify  X  as  a  Covariate  with  Y 

Procedure  —  Adjust  Y  to  account  for  X's 
effect  on  it .  Combines  ANOVA  and 
Regression  to  reduce  MSE 

With  ANCOVA  —  attribute  cause  of 
variation  to  appropriate  factor  -- 
diameter  in  Monty  4-4. 
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Alpha-Optimality 


□  Factorial  designs  are  (by  construction)  largely 
orthogonal 

□  These  are  a  class  of  designs  that  attempt  to 
optimize  information  from  an  experiment  without 
full  orthogonality 

□  You  specify  either:  model  to  fit  or  points  to  run 

□  D  Optimal:  a  D-optimal  design  will  maximize  the 
orthogonality  of  the  the  design  matrix  (the 
determinant) 

□  A-Optimal  maximizes  the  diagonal  while 
minimizing  the  off  diagonal  elements  of  the 
design  matrix 

□  Notes: 

■  Statistica  has  5  algorithms  to  search  for  best  design 

■  Most  useful  if  you  are  constrained  in  experimenting  (many 
infeasible  regions  or  required  points) 

■  Solutions  are  not  unique  or  guaranteed  to  be  optimal 
(search  algorithm  may  get  stuck  in  local  max/min) 

□  May  be  used  to  repair  or  augment  an  incomplete 
experiment  with  many  missing  data  points  -- 
which  small  subset  of  points  to  run  next. 
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Example  —  D-Optimal  Design 


Scatterplot  of  Design  Points 


2  factors;  20  runs;  D-optimal 


Model  incl.:  linear  effects/interacts 


MACHINE 


Machine 

Diam 

1 

20 

2 

30 

3 

25 

Force  these  points 


¥ 

¥ 

¥ 

¥ 
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¥ 

¥ 

¥ 

¥ 

¥ 
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¥ 

¥ 

¥ 

¥ 

¥ 

¥ 


Optimal  Design  Result:  Response  Sur&ce 


DESIGN  SUHHAPY: 

Type  of  design:  Response  surface  design 
Number  of  factors  or  components:  2 
Number  of  runs:  20;  selected  from  15 
Criterion:  D  (maxim,  det.  of  X'X-inverse) 
Algorithm:  Sequential  (Dykstra) 

Final  Statistics:  D=31641.2  [D-ef f iciency=66. 69%) 


^Display/Edit/Save  Design  = 


^isplay/print/save  design  |  Print  INFO  bow  | 


fp Denote  factors 
C  by  ^umbers 
C  by  Letters 
(*  by  Names 


p Order  of  runs  = 


f^^tandard  order  _ 

r  Random;  Seed:  1 6725668  E 


OK 


Cancel 


pAdd  to  the  design^ 


0  u^nuine  replicates  |o  ^  blank  columns 


Summary  of  efficiency  measures 


Correlation  matrix 


Inverse 


10^  2D  Scatterplot  of  points 


3D  Scatterplot  of  points 


□  Note  that  all  classical  designs 
optimal  by  these  criteria 
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Montgomery's  Critique  of 
Taguchi 


Taguchi  and  Traditional  Loss  Functions 


*3 

o 

C/3 


C/3 

C/3 

O 

h-] 


J 

/I 

‘~r 

1 

1 - ^ - 

-1  0  1 

Variance  around  nominal 


□  Taguchi's  quality  philosophy: 

■  minimum  variation  around  nominal 

■  robust  to  environmental  or  parameter 
variance 

□  Taguchi's  Design  philosophy 

■  interactions  are  uncommon  and 
unimportant 

□  Taguchi's  analysis  philosophy 

■  Use  S/N  ratio's  and  "pick  the  winner" 
marginal  means  plots 

□  Summary: 

■  Taguchi  quality  philosophy 
commendable 

■  Taguchi  design  and  analysis  practices 
inefficient,  ineffective,  and  needlessly 
complicated. 
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Taguchi  L9  Orthogonal  Array 


'is  Alias  Stm 

Lcture  (monty  4-4  ancova.sta) 

DESIGN 

Ef  feet 

b':' :  4  r 

*  =  par 

actors; 
tially  c 

all  rac 
Dr  compl 

-I 

tors  na^ 
etely  c( 

J  le^ 
□nf ounde 

^els 

id 

2 

1 

3 

1 

4 

z 

3 

z 

4 

J 

4 

1 

1 

* 

* 

* 

2 

* 

* 

3 

* 

* 

* 

4 

□  2  designs  —  Compare  alias  structures 

■  24-1  half  fraction  in  8  runs 

■  Taguchi  L9  4  factors  in  9  runs 

□  Conventional  fractional  factorials  or 
Plackett-Burman's  have  clearer  alias 
structures  than  Taguchi  OA's  and 
usually  a  foldover  or  complementary 
run  approach  to  continue 
experimentation 


'ifl  Aliasing  of  Effects  (Computed  from  Generators) 


EXPERIM 

DESIGN 

Factor 


2**(4-l)  design 

(Factors  are  denoted  by  numbers) 


34 


ji 

4 

12 

13 

23 

14 

24 

34 

234 

134 

124 

123 

34 

24 

14 

23 

13 

12 


misu 
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Design  of  Experiments 


|j  Nested  Designs  —  Formulatior 


¥ 


Variable  A: 

Supplier  Jones 

Supplier  Smith 

Variable  B:  Sanple  1 

Sairple2  Sanple3 

Sanple  4 

Sanple  1 

Sanple2  Sairple3 

Sarrple  4 

¥ 

¥ 

¥ 

P 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 

¥ 


Sample  5 


Sample  6 


Sample  8 


Sample  7 


□ 


□ 


□ 


□ 


□ 


Jones  Sample  1  o  Smith  Sample  1 

Levels  of  one  factor  (e.g.  B)  are  similar 
but  not  identical  for  levels  of  another 
factor  (e.g.  A). 

Montgomery's  example  —  purity  of  raw 
material  from  two  suppliers  —  three 
measurements  each. 

Statistica  example:  test  of  four  English 
classes  —  two  from  each  high  school 

Instead  of  crossed  (as  in  factorial)  B  is 
said  to  be  nested  in  variable  A 

If  factor  can  be  renumbered  as  in  purity 
example,  design  is  nested. 


Variable  A: 
Variable  B: 


Park  Hills  H.S. 

Fairborn  Baker  H.S. 

Ms.  Jones 

Ms.  Pahl 

Mr.  Frank 

Mr.  Ernest 
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Nested  Designs  —  Analysis 


P  Interpretation:  techniques  differ  in  effects  by  altitude,  major  differences 
gBB  between  operators  —  implies  select  technique  by  operator  and  altitude 


Variables 


Covariates 


B  OK 


Independent  (factors):  TECH_COD-ALT_M 
Dependent:  TRK_TIME 
Covariates:  none 

Codes  for  between-groups  factors:  |  Selected 

R^epeated  measures  design:  |  none 

N.ested  design: 


Cancel 


D 

7 

RESID3 

3. 

3. 

1. 

3. 

o 

2. 

4. 

2. 

3. 

2. 

4. 

0. 

2 

Sting  for  Between- Group  Factors 


Random  factors: 


isolated  control  group: 


:TECH_COD 
Nested  in: 


I”  R^ression  approach  (Type^P!_ 

For  large  main  effect  and  non  full-factorial  c 
with  unbalanced  nesting,  and  miKed-model 
Components  or  Experimental  Design  modul 


Levels; 


i5“i 


l^OPERATOR 
Not  Nested 

|TECH_COD 
ALT  M 


ALT  M 


Not  Nested 

TECH_COD 

OPERATOR 


Levels:  js  ^  |  Levels:  |2  ^ 


OK 


Cancel 


Specify  the 
neste 
and\ 
are  n\ 


IIAHUVA 


Ef  feet 

MS 

Ef  feet 

df 

Error 

MS 

Error 

F  X 

p- level 

6 

131 . 9444 

36 

3  .  277778 

40 . 25424\ 

.000000 

2 

2 

130 . 6667 

36 

3  .  277778 

39  .  86441 

.  000000 

3 

1 

50  .  0741 

36 

3  .  277778 

15  .  27684 

.  000393 

12 

— 

— 

— 

— 

— 

— 

13 

6 

20  .  8333 

36 

3  .  277778 

6  .  35593 

.  000125 

23 

2 

5  .  6296 

36 

3  .  277778 

1  .  71751 

.  193895 

123  “ 

— 

— 

— 

— 

— 

— 

lU 

i 

.  press  FI 


□ 


□ 


□ 


¥ 


In  balanced  nested  design,  there  are  a 
levels  of  A,  b  levels  of  B  nested  in  A 

Suppose  you  had  custom  techniques  for 
each  operator  in  an  ECM  design  — 
Technique  nested  in  Operator.  In 
ANOVA  module,  select  All  Codes  for 
each  factor  then  specify  Nesting 

Note  there  is  no  interaction  between  the 
nested  factors  (Operator  and 
Technique)  in  a  nested  design 


“F 
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Split  Plot  Designs  — 
Formulation 


Design  of  Exj^riments 


□ 


□ 


□ 

□ 


In  some  block  designs  we  may  not  completely 
randomize  run  order  within  blocks  --  perhaps  a 
split-plot  design. 

Consider  a  ground  mount  with  3  different 
power  sources  for  threat.  Wish  to  check  4 
ECM  techniques  for  each  power  source.  Want 
3  replicates  but  can  only  run  12  runs  in  our 
hour  of  testing  per  day. 

Execution:  set  power,  run  four  techniques. 
Change  power  source.  Run  four  techniques. 

Might  consider  this  a  factorial  with  days  as 
blocks  but  consider  restriction  on 
randomization 


Techniques 
randomized  within 
Power  source  — 
Each  Mission  block 
divided  into  three 
Power  whole  plots  - 
-  Techniques  are  a 
split  plot  treatment 
within  whole  plot. 
Impact  of  time- 
ary  ing  unknown? 


1 

Power 

Technique 

Mission 

Response 

1 

Commercial 

A 

1 

1 

B 

1 

1 

V  c 

1 

1 

N.  D 

1 

Diesel 

1 

1 

c\. 

1 

1 

D  ^ 

y  1 

1 

\l 

1 

Turbine 

1 

A  ^ 

1 

^  c 

1 

l\ 

1 

1 

Commercial 

C 

1 

D 

2^^ 

AF  T&.E  Days  -  Dec  05 


1-133 


u 


Split  Plot  Analysis 

Blocked  Factorial  ANOVA  Table 

A-TECH_COD  (3),  B-POWER_S  (3),  C-MSN_BUC  (2) 

df 


MS 


Source 

Name 

SS  Effect 

Effect 

Effect 

F 

p -level 

A 

Tech 

100.3 

2 

50.2 

7.6 

0.0143 

B 

Power 

76.0 

2 

38.0 

5.7 

0.0285 

AB 

138.7 

4 

34.7 

5.2 

0.0228 

C 

Blocks 

18.0 

1 

18.0 

~ 

Error 

Blk  X  Tr  Interaction 

53.0 

8 

6.625 

AC 

20.3 

2 

10.2 

BC 

12.0 

2 

6.0 

ABC 

20.7 

4 

5.2 

Total  386.0  17 

SpUt  Plot  ANOVA  Table 

A-TECH_COD  (3),  B-POWER_S  (3),  C-MSN_BLK  (2) 


Source 

Name 

SS  Effect 

df  Effect 

MS  Effect 

Spht  Plot  F 

Factorial  F 

Spht  Plot  p 

B 

Main  TR  Power 

76.0 

2 

38.0 

6.3 

5.7 

0.1364 

C 

Msn  Blocks 

18.0 

1 

18.0 

~ 

BC 

Main  Error 

12.0 

2 

6.0 

- 

A 

Subplot  TR  -  Tech 

100.3 

2 

50.2 

9.7 

7.6 

0.0292 

AB 

Main  x  Sub  (PxT) 

138.7 

4 

34.7 

6.7 

5.2 

0.0461 

AC 

Sub  xBlk 

20.3 

2 

10.2 

~ 

ABC 

Subplot  Error 

20.7 

4 

5.2 

~ 

Total  386.0  17 

□  In  blocked  factorial  --  blocks  interactions 
estimate  error 

□  In  Split-plot,  two  error  estimates  --  among 
whole  plots  and  within  subplots 

□  keep  this  model  in  mind  when  factors  are 
difficult  to  change  and  you  restrict 
randomization  within  blocks.  May  cause  you 
to  re-analyze  data  as  split  plot. 
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P 

P 

P 
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P 

P 

P 


Fixed  vs.  Random  Effects 
Model  Formulation 


Histogram  (random  effects.STA  10v*49c) 


RWR  angle  error  —  fixed  X 


<=20  (40,60]  (80,100]  (120,140] 

(20,40]  (60,80]  (100,120]  (140,160] 

AONBIN 


□  In  fixed  effects  model,  only  Y  is  a  random 
variable  —  we  choose  and  fix  X  at  a  levels 

□  In  random  effects  model,  we  randomly 
choose  a  levels  of  X  from  a  large  number 
of  values 

□  In  random  effects  model,  both  Y  and  X  are 
random 

□  In  mixed  model,  some  X  are  fixed,  some 
random 

□  Example  —  actual  angle  when  df  error 
measured  in  RWR  tests.  Angle  is  a  random 
factor  rather  than  same  angle  each  pass 


Histogram  (random  effects.STA  10v*49c) 


RWR  angle  error  —  random  X 

<=0  (20,40]  (60,80]  (100,120]  (140,160] 

(0,20]  (40,60]  (80,100]  (120,140]  >160  _ 
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Random  Effects 
Statistical  Model 


Design  of  Experiments 
#  - - ^  ^ 


■-€  ^  r 


Model:  ~ 


+  SS 

Treatments  Error 


y 

Fixed  Effects  Model 

V{y,;)  =  o'^ 


Tii  +  T  + 


IJ 


Random  Effects  Model 


o 

II 

•  • 

Ho:a,"  =  0 

O 

•  • 

a 

i 

^  Treatments  ^ 

a-\ 

=  <j^  +  na 

E(MSE)  =  cr" 

p  _  ^Treatments 


MS 


Error 


□  Random  effects  also  known  as 
Components  of  Variance  model  —  part  to 
X,  part  to  Error 

□  Inferences  concerning  one  level  of  X 
meaningless; 

□  Inferences  apply  to  entire  population  of  X 
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Random  Effects  Model  — 
Analysis 


Design  of  Experiments 


1 

MAIN  EFFECT:  AONTRUNC  (random  effects.sta) 

1 

Error:  With 

lin 

1 

From:  Means  (unweighted)  (random  eff 

F(34,3)=95.04;  p<.0015 

60  . 

Sum  of 
Squares 

df 

Mean 

Square 

F 

p-level 

Effect 

13549.72 

34.00 

398.5213 

95.04 

0.0015 

/in  _ 

Error 

12.57942 

3.00 

4.193141 

oo 


-I-  SSj; 

H„:  CT,  =  0 

Ha:  <5x>0 


0  10  20  30  40  50  60  70  80  90  100  110  120  130  140  150  160  170  180 

5  15  25  35  45  55  65  75  85  95  105  115  125  135  145  155  165  175 


Angle  Off  The  Nose  (Random  X) 


r 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1  □  We  must  account  for  both  variance  of  X  and  the 

1  variance  of  Y  in  our  model 

1  □  No  interaction  between  predictors  when  X  is 

1  random  or  mixed 

1  □  No  estimate  of  individual  cell  means  for  levels 

of  X 

1  □  Simply  --  are  some  levels  of  X  different  than 

others? 

1 

1 

Summary  of  all  Analyzed  as  a  fixed  effects  model 

1 

1-AONCODE,  2-t>IUbUUUb 

i 

1 

df 

MS 

df 

MS 

i 

Effect 

Effect 

Error 

Error 

F 

p-level 

1 

1 

1 

6 

623.21 

28 

55.82 

11.16 

0.000002 

w 

I 

i 

I 
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Intro  to  Response  Surfaces 


□ 


□ 


□ 


Response  Surface  Methodology  —  a 
collection  of  math/stat  tools  to  optimize 
settings  of  quantitative  predictor  (X) 
variables  on  Y 

Y  =  f  (Xi,  X2,  ...Xk)  +s 

Example  --  effect  of  temperature  and 
pressure  on  chemical  process  or  effect 
of  scan  rate  offset  and  duty  cycle  on 
track  errors 

General  procedure  is 

■  start  at  a  region 

■  estimate  first  order 

■  climb 

■  center  second  matrix 

□  If  curvature,  discover  min/max  points 

□  If  not  —  climb  again  and  iterate 
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Candidates  for  RSM 


□  Quantitative  predictors  and  response 
variables 

□  some  indication  of  gradients  to  exploit 

□  When  an  empirical  model  is  desired 

□  Start  with  as  many  variables  as  desired 

□  Cull  to  less  than  6  predictor  variables 
(interactions  are  harder)  —  best  if  3  or  4 

□  In  my  Wing 

■  Modeling  and  Simulation 

■  HWIL  (EW  Ground  mounts) 

■  AFEWES  and  other  installed  test  fac'y 

■  Sys  Integration  Lab  testing 
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Typical  RSM  Models 


□ 


□ 


□ 


First  Order  Model 
y  =  Po  +  +  P2X2  + ...  +  P^x^  +  8 

Second  Order  Model 

r\  r\ 

y  =  PO  +  P^Xy  +  Py 


First  order  model  for  first 
check  of  region  (with  center 
points) 

Use  center  points  to  estimate 

■  interactions, 

■  errors,  and 

■  curvature 

If  needed,  fit  RSM  model  (CCD 
or  B-B)  to  local  region 


2^  with n=4  cent 


Central  Composite 
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Design  of  Experiments 
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Method  of  Steepest  Ascent 


X. 


Ymax 


Region  of 
first  order 
fit 


□ 


□ 


Steps  along  the  path  are  proportional  to 
the  first  derivative  of  the  first  order 
model  (slopes  of  the  variables) 

Method  usually  uses  coded  variables  and 
transforms  back  and  forth 


Transform  Coded 


X 


coded 


X  —X 

n  n 


X  —X 

max  mm 


Step  size  x.  =  Ax.  from  Max  Pj 
Steepest  Ascent  Units 
Pi 


Ax,  = 
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Second  Order  Design 
Properties 


Design  of  Experiments 
#  - - ^  ^ 


□ 


We  Want 

■  reasonable  distribution  of  data 
points  for  robustness  and  scope 

■  allows  model  adequacy  including 
curvature  and  lack  of  fit  to  be 
investigated 

■  ailows  experiments  in  blocks 

■  allows  higher-order  designs  to  be 
built  up  sequentially 


I 

i 

i 

i 

P 

P 

P 

P 

P 

P 

P 


■  provides  an  internal  estimate  of 
error 

■  requires  as  few  points  as  possible 

■  enough,  but  not  too  many  levels 
of  the  independent  variabie 

■  ensures  simplicity  of  caiculations 

■  is  orthogonal 

■  is  rotatable 
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Second  Order  Orthogonal 
I  Designs 
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Design  of  Exj^riments 


Design  of  Elxperiments 

'9 


Central  Composite 


3 

M 


Box  Behnken 
Spherical  Edge- 
Centered  Design 


Matrr 


f 
I 
P 
P 
P 
P 
P 
P 


□UUn 


ni^i  /n  n^ii  ni  iiimho - CAt — i — ota  h  >i.  .*/i 


8  Ball  RSM  with  CCD  in  4  Variables 


Face-Centered 

CCD 


.DD D, 


=DD D, 
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The  Least  You  Should  Know...®! 


□  3-level  designs  seem  attractive 
when  curvature  is  desired 

□  Drawbacks  include: 

■  larger  designs  (3'^) 

■  difficult  to  interpret  quadratic 
interactions 

■  undesirable  fractional  confounding 
patterns 

□  Summary  —  3'<  designs  are  a 
"solution  looking  for  a  problem " 


□  Mixed  2  and  3  level  designs  can 
be  represented  in  2k  model 

■  may  need  to  roll  own  mixed  design 
and  let  Statistica  analyze  it 

□  Mixed  models  with  four  level 
variables  can  be  represented  with 
2x2-level  variables 
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The  Least  You  Should  Know... 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


Variations  on  factorial  designs  and  ANOVA 
analysis  are  many 

You  know  the  mainline  approaches  and 
have  the  math  to  learn  others 

If  B,  differs  with  A  settings,  you  have  a 
nested  design 

Split  plot  designs  analyze  restricted 
randomization  within  blocks 

If  X  is  chosen  (or  observed)  randomly,  the 
model  is  a  random  effects  model 

When  X  is  uncontrollable,  affects  Y,  and 
can  be  observed,  it  may  be  a  covariate 
analyzed  by  ANCOVA 

Alpha-optimality  (D-,  A-,  G-)  assumes 
substantial  process  knowledge.  May  be 
used  for  repair  or  constrained  situations 

Taguchi  made  strong  contributions  to 
quality  improvement;  there  are  better 
DOE  approaches  than  Taguchi's,  however. 
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The  Least  You  Should  Know...®! 


□  RSM  is  a  straight-forward  extension 
of  factorial  designs  --  with  a 
navigation  tool 

□  Good  RSM  candidates  have  3-5 
continuous  variables  with  a 
gradient  to  exploit 

□  Basic  RSM  method: 

■  1st  order  factorials  with  cp  to  estimate 
gradient 

■  ascend  gradient  to  top  (15-20  runs) 

■  run  factorial  with  cp's  to  detect 
curvature  near  optimum 

■  augment  an  RSM  design  to  estimate 
model 

■  Confirm  optimum  with  factorial  near 
optimal  value 

□  Consider  either  central  composite 
design  or  Box-Behnkin  design  for 
second  order  models 
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